Running AI locally sounds difficult.
It isn't.
Twenty minutes from now you'll have a real AI assistant running entirely on your laptop. No API keys. No cloud account. No monthly bill. Just Ollama, a lightweight language model, and Hermes.
Who Is This For?
This guide is for you if:
- You're a JavaScript developer
- You're AI-curious but haven't run a local model
- You want to experiment without paying for API calls
- You'd rather learn by building than by reading whitepapers
Sound familiar? Let's go.
The Pieces
Ollama runs models. Think of it like PostgreSQL for language models — you install it, start it, and it runs a local server in the background. It handles downloading, GPU memory, and exposes a clean REST API.
The model is the brain. An open-source LLM that Ollama downloads and serves. Switching models is as simple as downloading another one and changing a single config value. Your application barely notices.
Hermes is an autonomous agent layer from Nous Research — it edits files, runs shell commands, browses the web, and keeps persistent memory across sessions, all through tool calls to whatever model you point it at. It's a heavier lift than a simple chat wrapper, so we'll set it up carefully.
Now let's build one.
The Setup
1. Install Ollama
On a Mac:
brew install ollama
On Linux or Windows, grab the installer from ollama.com. One download, no sign-up, no configuration wizard.
Start the service:
ollama serve
This runs in the background. Leave it going.
2. Download Your First Model
I'm starting you with a small model on purpose. Here's why: I want your first experience to succeed. Once you know everything works, upgrading to a larger model is one command. But let's get a win first.
ollama pull gemma3:4b
This downloads in about two minutes.
3. Verify It Works
ollama run gemma3:4b
You'll drop into an interactive prompt. Type something.
>>> Write a JavaScript function that takes an array and returns the three largest numbers.
If you get a response, you're running a local LLM. No API key. No internet required after the download.
This is the moment where it clicks.
4. Size Up for Agent Work
Chatting and running an agent are different jobs.
The 4B model you just verified is great for chat — fast, lightweight, and it proves Ollama works. But Hermes needs a model that reliably follows tool-call instructions. Smaller models (3B–7B) sometimes ignore those instructions entirely and just respond in plain text instead of taking the action.
For agent work, step up:
ollama pull gemma3:12b
Yes, it's larger. Yes, it needs more RAM. But it's the difference between an assistant that actually edits your files and one that just talks about editing them.
Check the hardware reference below if you're unsure whether your machine can handle it.
5. Install Hermes
Hermes is a separate install from Ollama. Run the official installer:
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
This drops Hermes into ~/.hermes/ and walks you through a setup wizard on first run.
6. Fix the Context Window Before You Do Anything Else
This is the step almost everyone misses, and it will make Hermes behave erratically if you skip it. Hermes requires a minimum 64,000-token context window to hold its working memory across multi-step tool calls. Ollama defaults to 4,096. If you don't fix this first, Hermes will start fine and then fall apart after 3-4 tool calls.
Bump the context on your model with a Modelfile:
cat > Modelfile << 'EOF'
FROM gemma3:12b
PARAMETER num_ctx 64000
EOF
ollama create gemma3-12b-64k -f Modelfile
Verify it stuck:
ollama run gemma3-12b-64k --num_ctx 64000 "reply with just: OK"
ollama ps
Check the CONTEXT column — it should show 64000. If it still shows 4096, the Modelfile approach is the reliable fallback; don't rely on environment variables alone.
7. Connect Hermes to Ollama
When the Hermes setup wizard asks for a provider, choose the custom OpenAI-compatible endpoint option:
- Endpoint:
http://localhost:11434/v1 - API key: leave blank (Ollama doesn't require one)
- Model: the exact name from
ollama list— in our case,gemma3-12b-64k
Hermes writes this to ~/.hermes/config.yaml automatically. That's it — you now have a local, agentic AI assistant.
Switching Models
This is the killer feature. Ollama makes model-switching trivial.
Need something faster for chat?
ollama pull qwen2.5:3b
Need stronger reasoning?
ollama pull gemma3:12b
Need coding assistance?
ollama pull codellama:7b
Same API. Same workflow. Different model. One string in your config flips from gemma3:12b to codellama:7b and suddenly you have a coding specialist running locally.
Remember: If you plan to use a new model with Hermes, rebuild it with the 64K context Modelfile first. Skipping this is the most common reason Hermes falls apart after a few tool calls.
When You Outgrow Your Laptop
Local models have limits. If you're running on a machine with 8GB of RAM, you'll hit a ceiling with models larger than about 7B parameters. Your laptop fan will sound like a jet engine. And some models simply won't fit.
When that happens, you can always move to cloud-hosted models while keeping a similar architecture. The patterns you learn here — REST APIs, model switching, prompt engineering — carry over directly. You just swap out the endpoint.
Quick reference for what your hardware can handle:
- A 4B model needs roughly 3GB free RAM
- A 7B model wants 6GB
- A 13B model wants 10GB+
- Apple Silicon handles local inference well; older Intel Macs will struggle
- Local model context windows are usually 4K-8K tokens by default — shorter than cloud models, and something you'll need to raise manually for agentic tools like Hermes
For development, iteration, and experimentation, a 4B model running on your machine is more than enough for chat. Agent work is where you pay the RAM tax.
Wiring It Into Your Own Code
Ollama exposes a REST API at http://localhost:11434. It runs a local server, just like Express. You send it HTTP requests, just like any REST API.
Here is a minimal Express endpoint that calls your local model:
const express = require('express');
const app = express();
app.use(express.json());
app.post('/ask', async (req, res) => {
const response = await fetch('http://localhost:11434/api/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: 'gemma3:4b',
prompt: req.body.prompt,
stream: false
})
});
const data = await response.json();
res.json({ reply: data.response });
});
app.listen(3000, () => console.log('Local AI server on :3000'));
Twenty lines. Since Ollama already speaks REST, using it from Express feels just like calling any other local service. No SDK required.
What To Do When It Breaks
It will break. Here is what to check:
- Model won't load? You probably ran out of RAM. Close Chrome. Close VS Code. Try again.
- Slow responses? On a Mac with Apple Silicon, Ollama should use the GPU automatically. Verify with
ollama ps. - Hermes losing track after a few steps? Check that your context window is actually set to 64K — this is the single most common cause.
- Weird output? Local models are more sensitive to prompt structure than cloud models. Be more explicit. Give it examples. Tell it exactly what format you want.
This is the trade-off. A little polish for total control. After a week, you stop noticing.
The Real Win
The biggest surprise wasn't how good the model was. It was how easy it was to get running.
Twenty minutes later I had a local AI assistant I could experiment with as much as I wanted. No API key. No usage dashboard. No worrying about how many prompts I'd burned through. Just another service running on my machine, ready whenever I needed it.
If you've been putting off local AI because it looked intimidating, stop waiting. Twenty minutes is all it takes to get your first assistant running.
