# Local Model Setup > Set up local LLM providers (LM Studio, Ollama, llama.cpp) as Hermes Agent backends. - Skill: `wcpaka-lgtm/local-model-setup` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add wcpaka-lgtm/local-model-setup` - Raw SKILL.md: https://api.skillmd.com/api/skills/wcpaka-lgtm/local-model-setup/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: wcpaka-lgtm (https://skillmd.com/u/wcpaka-lgtm) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/wcpaka-lgtm/local-model-setup --- # Local Model Setup Connect Hermes Agent to locally-running LLMs via OpenAI-compatible API servers. Covers LM Studio (GUI + headless `lms`), Ollama, and llama.cpp server. ## Quick Decision Matrix | Provider | Install | Start Server | Default Port | Model ID Source | |----------|---------|-------------|-------------|-----------------| | LM Studio (headless) | `irm https://lmstudio.ai/install.ps1 \| iex` | `lms server start` | 1234 | `lms ls` | | LM Studio (GUI) | Download from lmstudio.ai | Start Server button in UI | 1234 | GUI model list | | Ollama | `curl -fsSL https://ollama.com/install.sh \| sh` | `ollama serve` | 11434 | `ollama list` | | llama.cpp | Build from source | `./llama-server -m model.gguf` | 8080 | manual | ## Common Setup Pattern (Hermes Config) All local providers use the `custom:` provider pattern with no API key: ```bash hermes config set model.provider "custom:" hermes config set model.base_url "http://localhost:/v1" hermes config set model.default "" hermes config set model.context_length hermes config set model.api_key "" ``` **Config changes take effect on the next session** — restart Hermes or `/reset` to activate. --- ## LM Studio (Headless `lms` CLI) — Preferred on Windows The GUI installer's `/S` (silent) flag is unreliable on Windows. Use the headless CLI instead. ### Install ```powershell # Windows (PowerShell) irm https://lmstudio.ai/install.ps1 | iex ``` ```bash # macOS / Linux curl -fsSL https://lmstudio.ai/install.sh | bash ``` Binary lands at `~/.lmstudio/bin/lms`. ### Check Available Models ```bash lms ls ``` Model IDs follow `org/model-name` format (e.g., `google/gemma-4-12b`). ### Download a Model (if needed) ```bash lms get google/gemma-4-12b ``` ### Start the API Server ```bash lms server start --port 1234 ``` The server provides an OpenAI-compatible API at `http://localhost:1234/v1`. Verify with: ```bash curl -s http://localhost:1234/v1/models ``` ### Hermes Config Commands ```bash hermes config set model.provider "custom:lmstudio" hermes config set model.base_url "http://localhost:1234/v1" hermes config set model.default "google/gemma-4-12b" hermes config set model.context_length 131072 hermes config set model.api_key "" ``` --- ## Reasoning / Thinking Models Some models (Gemma 4, DeepSeek R1, QwQ) use a reasoning/thinking mode where the model emits internal reasoning tokens before the final answer. In the API response, this appears as `reasoning_content` in `choices[0].message` with an empty or delayed `content` field. **Detection**: If a test call returns `finish_reason: "length"` with empty `content` but populated `reasoning_content`, the model is a reasoning model. **Fix**: Enable reasoning in Hermes: ```bash hermes config set model.reasoning_effort "high" ``` Available levels: `none`, `minimal`, `low`, `medium`, `high`, `xhigh`. **DeepSeek V4 effort mapping (verified live 2026-08 via opencode-go relay)**: the opencode-go provider profile maps Hermes effort to the wire — `max`/`ultra`/`xhigh` → `reasoning_effort:"max"`, `low`/`medium`/`high` → same value, `none` → `thinking:{type:disabled}`. Official DeepSeek docs (api-docs.deepseek.com/guides/thinking_mode) map requested→actual: **Flash**: low→low, high→high, xhigh→high (capped), max→max; **Pro**: low→high, high→high, xhigh→max, max→max. So `reasoning_effort: max` IS accepted by deepseek-v4-flash — do not trust "official doesn't support max" claims without testing; send `reasoning_effort:max` and check it's accepted. Thinking mode silently ignores temperature/top_p/presence/frequency penalties. In tool-call turns the assistant's `reasoning_content` MUST be echoed back in later requests or the API 400s. --- ## Ollama ### Install & Start ```bash # macOS / Linux curl -fsSL https://ollama.com/install.sh | sh # Start (usually auto-starts as service) ollama serve ``` ### Pull & Verify ```bash ollama pull llama3.2:3b ollama list ``` ### Hermes Config ```bash hermes config set model.provider "custom:ollama" hermes config set model.base_url "http://localhost:11434/v1" hermes config set model.default "llama3.2:3b" hermes config set model.context_length 131072 hermes config set model.api_key "" ``` --- ## Custom llama.cpp forks (non-mainline GGUFs) Some new-model GGUFs only load in a project-specific llama.cpp fork (the `maple` architecture for Maple-Preview lives only in `stamsam/llama.cpp` branch `prism`; mainline llama.cpp AND Ollama refuse the file). ALWAYS read the GGUF model card for the exact fork URL + branch + commit BEFORE downloading — never assume mainline works. Windows build (CPU-only — no CUDA toolkit needed; MSVC via the VS generator): ```bash git clone --branch --single-branch && cd cmake -B build -G "Visual Studio 17 2022" -A x64 -DLLAMA_CUDA=OFF cmake --build build --config Release --target llama-cli llama-server llama-bench llama-completion # binaries land in build/bin/Release/ ``` Single-shot non-interactive runs may need the fork's dedicated binary (`llama-completion`) — `llama-cli` in some forks drops `-no-cnv` and forces chat mode that ignores `-n`. Full Maple-Preview recipe (fork rev, GGUF download, real speed numbers, test methodology): `references/maple-preview-windows-setup.md`. ## Verification After config changes and restart: ```bash hermes status # Check Model and Provider lines hermes doctor # Full diagnostics ``` Quick API test (before Hermes restart): ```bash curl -s http://localhost:/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"","messages":[{"role":"user","content":"Say hello"}],"max_tokens":30}' ``` --- ## Hardware feasibility check (before recommending a model/engine) When the user asks "can my PC run locally?" (viral posts like "284B in 5.3 GB"), verify in this order instead of promising: 1. Engine CUDA arch support vs the user's GPU (Pascal sm_61 is the cutoff — GB10/L40S-targeted engines like antirez/ds4 CUDA won't run; llama.cpp-family usually still builds). 2. Memory path: does the engine SSD-stream, and on which backends? Streaming is often Metal/ROCm-only; CUDA paths may require full residency + multi-GPU. 3. GGUF size vs RAM+VRAM before quoting speeds. 4. Fallback offer: 2-bit asymmetric-quant SMALL MoE models (Qwen 3.6 35B-A3B ≈1.5 GB, Gemma 4 26B-A4B ≈2 GB) run great on old GPUs via Ollama and often beat a 2-bit-quantized 284B in quality. Full case study (DeepSeek V4 Flash 284B on a 1080 Ti — what works, what doesn't, measured speeds): `references/deepseek-v4-flash-local-feasibility.md`. ## Pitfalls 1. **GUI `lms` on Windows**: The GUI installer's `/S` silent flag often doesn't work. Always prefer the CLI `lms` via the PowerShell install script. 2. **LM Studio GUI vs CLI conflict**: Only one process can bind to port 1234. If the GUI is already running a local server, `lms server start` will fail with a port conflict. 3. **Reasoning models**: Gemma 4, DeepSeek R1, QwQ, and similar reasoning models will appear broken (empty responses, early cutoff) without `reasoning_effort` set. Always test with a simple curl call first before configuring Hermes. 4. **Config activation**: `hermes config set` writes to disk but the running session doesn't re-read config. Restart Hermes or use `/reset` in the CLI. 5. **API key for custom providers**: Set `model.api_key` to an empty string (`""`), not omitted. Omitting it may cause Hermes to look for a default key. 6. **Context length**: Always set `model.context_length` explicitly. The default may be too low for the model. 7. **MSYS paths break native Windows exes (hit 2026-08)**: `llama-cli -m /c/Users/.../model.gguf` fails with "No such file or directory". Convert with `MODEL=$(cygpath -w "$HOME/path/model.gguf")` and pass the Windows path. 8. **Korean/UTF-8 args get CP949-mangled (hit 2026-08)**: Korean passed inline to a native exe (or inside `curl -d '...한글...'`) arrives garbled — the model sees mojibake, or the server rejects the JSON with 500 "ill-formed UTF-8". Write prompts to a UTF-8 file and use `-f file.txt` (`--prompt-file`); write JSON payloads to UTF-8 files and `curl -d @file`. Never pass Korean inline. 9. **`curl -d @~` does not expand `~`**: use `-d @$(cygpath -w ~/file)`. 10. **`cmd //c` gets mangled by git-bash** (opens an interactive shell instead of running the bat): use `MSYS_NO_PATHCONV=1 cmd /c script.bat`. 11. **Reasoning models eat the whole token budget (hit 2026-08)**: with small `max_tokens` (e.g. 80) ALL tokens go to `reasoning_content` and `content` returns null with `finish_reason: "length"` — looks like a broken model. Use generous budgets (300+) for reasoning models and check `reasoning_content` before concluding failure. ## References - `references/gemma4-lmstudio-session.md` — Full transcript of the Gemma 4 12B + LM Studio setup session - `references/maple-preview-windows-setup.md` — Maple-Preview (20B-A1B ternary) on Windows: fork build, GGUF download, real speed numbers, test methodology - `references/deepseek-v4-flash-local-feasibility.md` — DeepSeek V4 Flash 284B on a 1080 Ti: engine arch support, SSD-streaming backend limits, measured speeds