Local Model Setup

Local Model Setup

Connect Hermes Agent to locally-running LLMs via OpenAI-compatible API servers. Covers LM Studio (GUI + headless lms), Ollama, and llama.cpp server.

Quick Decision Matrix

Common Setup Pattern (Hermes Config)

All local providers use the custom:<name> provider pattern with no API key:

hermes config set model.provider "custom:<name>"
hermes config set model.base_url "http://localhost:<port>/v1"
hermes config set model.default "<model-id>"
hermes config set model.context_length <tokens>
hermes config set model.api_key ""

Config changes take effect on the next session — restart Hermes or /reset to activate.

LM Studio (Headless lms CLI) — Preferred on Windows

The GUI installer's /S (silent) flag is unreliable on Windows. Use the headless CLI instead.

Install

# Windows (PowerShell)
irm https://lmstudio.ai/install.ps1 | iex

# macOS / Linux
curl -fsSL https://lmstudio.ai/install.sh | bash

Binary lands at ~/.lmstudio/bin/lms.

Check Available Models

lms ls

Model IDs follow org/model-name format (e.g., google/gemma-4-12b).

Download a Model (if needed)

lms get google/gemma-4-12b

Start the API Server

lms server start --port 1234

The server provides an OpenAI-compatible API at http://localhost:1234/v1.

Verify with:

curl -s http://localhost:1234/v1/models

Hermes Config Commands

hermes config set model.provider "custom:lmstudio"
hermes config set model.base_url "http://localhost:1234/v1"
hermes config set model.default "google/gemma-4-12b"
hermes config set model.context_length 131072
hermes config set model.api_key ""

Reasoning / Thinking Models

Some models (Gemma 4, DeepSeek R1, QwQ) use a reasoning/thinking mode where the model emits internal reasoning tokens before the final answer. In the API response, this appears as reasoning_content in choices[0].message with an empty or delayed content field.

Detection: If a test call returns finish_reason: "length" with empty content but populated reasoning_content, the model is a reasoning model.

Fix: Enable reasoning in Hermes:

hermes config set model.reasoning_effort "high"

Available levels: none, minimal, low, medium, high, xhigh.

DeepSeek V4 effort mapping (verified live 2026-08 via opencode-go relay): the opencode-go provider profile maps Hermes effort to the wire — max/ultra/xhigh → reasoning_effort:"max", low/medium/high → same value, none → thinking:{type:disabled}. Official DeepSeek docs (api-docs.deepseek.com/guides/thinking_mode) map requested→actual: Flash: low→low, high→high, xhigh→high (capped), max→max; Pro: low→high, high→high, xhigh→max, max→max. So reasoning_effort: max IS accepted by deepseek-v4-flash — do not trust "official doesn't support max" claims without testing; send reasoning_effort:max and check it's accepted. Thinking mode silently ignores temperature/top_p/presence/frequency penalties. In tool-call turns the assistant's reasoning_content MUST be echoed back in later requests or the API 400s.

Ollama

Install & Start

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start (usually auto-starts as service)
ollama serve

Pull & Verify

ollama pull llama3.2:3b
ollama list

Hermes Config

hermes config set model.provider "custom:ollama"
hermes config set model.base_url "http://localhost:11434/v1"
hermes config set model.default "llama3.2:3b"
hermes config set model.context_length 131072
hermes config set model.api_key ""

Custom llama.cpp forks (non-mainline GGUFs)

Some new-model GGUFs only load in a project-specific llama.cpp fork (the maple architecture for Maple-Preview lives only in stamsam/llama.cpp branch prism; mainline llama.cpp AND Ollama refuse the file). ALWAYS read the GGUF model card for the exact fork URL + branch + commit BEFORE downloading — never assume mainline works.

Windows build (CPU-only — no CUDA toolkit needed; MSVC via the VS generator):

git clone --branch <branch> --single-branch <fork-url> && cd <dir>
cmake -B build -G "Visual Studio 17 2022" -A x64 -DLLAMA_CUDA=OFF
cmake --build build --config Release --target llama-cli llama-server llama-bench llama-completion
# binaries land in build/bin/Release/

Single-shot non-interactive runs may need the fork's dedicated binary (llama-completion) — llama-cli in some forks drops -no-cnv and forces chat mode that ignores -n.

Full Maple-Preview recipe (fork rev, GGUF download, real speed numbers, test methodology): references/maple-preview-windows-setup.md.

Verification

After config changes and restart:

hermes status                    # Check Model and Provider lines
hermes doctor                    # Full diagnostics

Quick API test (before Hermes restart):

curl -s http://localhost:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Say hello"}],"max_tokens":30}'

Hardware feasibility check (before recommending a model/engine)

When the user asks "can my PC run locally?" (viral posts like "284B in 5.3 GB"), verify in this order instead of promising:

1. Engine CUDA arch support vs the user's GPU (Pascal sm_61 is the cutoff — GB10/L40S-targeted engines like antirez/ds4 CUDA won't run; llama.cpp-family usually still builds).
2. Memory path: does the engine SSD-stream, and on which backends? Streaming is often Metal/ROCm-only; CUDA paths may require full residency + multi-GPU.
3. GGUF size vs RAM+VRAM before quoting speeds.
4. Fallback offer: 2-bit asymmetric-quant SMALL MoE models (Qwen 3.6 35B-A3B ≈1.5 GB, Gemma 4 26B-A4B ≈2 GB) run great on old GPUs via Ollama and often beat a 2-bit-quantized 284B in quality. Full case study (DeepSeek V4 Flash 284B on a 1080 Ti — what works, what doesn't, measured speeds): references/deepseek-v4-flash-local-feasibility.md.

Pitfalls

1. GUI lms on Windows: The GUI installer's /S silent flag often doesn't work. Always prefer the CLI lms via the PowerShell install script.

2. LM Studio GUI vs CLI conflict: Only one process can bind to port 1234. If the GUI is already running a local server, lms server start will fail with a port conflict.

3. Reasoning models: Gemma 4, DeepSeek R1, QwQ, and similar reasoning models will appear broken (empty responses, early cutoff) without reasoning_effort set. Always test with a simple curl call first before configuring Hermes.

4. Config activation: hermes config set writes to disk but the running session doesn't re-read config. Restart Hermes or use /reset in the CLI.

5. API key for custom providers: Set model.api_key to an empty string (""), not omitted. Omitting it may cause Hermes to look for a default key.

6. Context length: Always set model.context_length explicitly. The default may be too low for the model.

7. MSYS paths break native Windows exes (hit 2026-08): llama-cli -m /c/Users/.../model.gguf fails with "No such file or directory". Convert with MODEL=$(cygpath -w "$HOME/path/model.gguf") and pass the Windows path.

8. Korean/UTF-8 args get CP949-mangled (hit 2026-08): Korean passed inline to a native exe (or inside curl -d '...한글...') arrives garbled — the model sees mojibake, or the server rejects the JSON with 500 "ill-formed UTF-8". Write prompts to a UTF-8 file and use -f file.txt (--prompt-file); write JSON payloads to UTF-8 files and curl -d @file. Never pass Korean inline.

9. curl -d @~ does not expand ~: use -d @$(cygpath -w ~/file).

10. cmd //c gets mangled by git-bash (opens an interactive shell instead of running the bat): use MSYS_NO_PATHCONV=1 cmd /c script.bat.

11. Reasoning models eat the whole token budget (hit 2026-08): with small max_tokens (e.g. 80) ALL tokens go to reasoning_content and content returns null with finish_reason: "length" — looks like a broken model. Use generous budgets (300+) for reasoning models and check reasoning_content before concluding failure.

References

• references/gemma4-lmstudio-session.md — Full transcript of the Gemma 4 12B + LM Studio setup session
• references/maple-preview-windows-setup.md — Maple-Preview (20B-A1B ternary) on Windows: fork build, GGUF download, real speed numbers, test methodology
• references/deepseek-v4-flash-local-feasibility.md — DeepSeek V4 Flash 284B on a 1080 Ti: engine arch support, SSD-streaming backend limits, measured speeds