Local LLM Serving
Class-level runbook for getting a self-hosted LLM serving stack running on a workstation: controller Hono API, Next.js frontend, llama.cpp backend, model download, recipe registration, launch, and end-to-end verification through the OpenAI-compatible proxy.
When to use this
- The user wants to run a local LLM stack (Local Studio or similar controller + frontend + inference backend).
- You need to go from "backend installed" to "chat completion works and usage is recorded."
- The machine is CPU-only or GPU-less, and you need a recipe that avoids trying to use GPU.
Standard flow
Check controller status
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/api/system-status
If the controller isn't running, start it from the controller directory first.
Install the inference backend
Download a model
Use the controller's download endpoint; pick a small GGUF for CPU (e.g., Qwen2.5-0.5B-Instruct Q4_K_M):
curl -s -X POST http://127.0.0.1:8080/studio/downloads \
-H 'Content-Type: application/json' \
-d '{"model_id":"bartowski/Qwen2.5-0.5B-Instruct-GGUF","allow_patterns":["*Q4_K_M.gguf"]}'
Then confirm with GET /v1/studio/models.
Create a recipe
Recipes are the controller's unit of model launch configuration. See templates/llamacpp-cpu-recipe.json for a known-good CPU-only starter. Required fields:
id, name, model_path (absolute)
backend: llamacpp
runtime: { "kind": "binary", "ref": "/usr/local/bin/llama-server" }
host, port, served_model_name
extra_args: { "n-gpu-layers": 0 } for CPU-only machines
Send it to POST /recipes.
Launch the model
curl -s -X POST http://127.0.0.1:8080/launch/<recipe-id>
curl -s "http://127.0.0.1:8080/wait-ready?timeout=120"
Verify the OpenAI-compatible surface
curl -s http://127.0.0.1:8080/v1/models
curl -s -X POST http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<served_model_name>","messages":[{"role":"user","content":"hello"}],"max_tokens":50}'
Check usage accounting
curl -s http://127.0.0.1:8080/usage?include_controller=true
Key route map
| Endpoint |
Purpose |
GET /health |
Controller liveness |
GET /api/system-status |
Controller + inference status |
GET /runtime/targets |
Available backends/runtimes |
POST /studio/downloads |
Download a model from Hugging Face |
GET /v1/studio/models |
Registered models |
POST /recipes |
Register a launch recipe |
POST /launch/:recipeId |
Launch a recipe |
GET /wait-ready |
Wait for inference backend healthy |
GET /v1/models |
OpenAI-compatible model list |
POST /v1/chat/completions |
OpenAI-compatible chat completion |
GET /usage |
Inference usage stats |
GET /usage?include_controller=true |
Include controller request telemetry |
Windows GPU + MoE offloading
For MoE models (Laguna-XS-2.1, Qwen3.6 MoE) on Windows with CUDA, combine
-ngl (layer-level GPU offload) with -cmoe (MoE expert weights to CPU).
These are independent mechanisms — -ngl controls shared weights, -cmoe
controls expert tensors via regex. See references/windows-gpu-moe-offloading.md
for the full runbook, build commands, and VRAM budget.
Choosing a model size for the machine
Before recommending a model, probe the actual hardware — don't guess from the model name. Run the probe commands in references/model-sizing-constrained-mac.md, then size by this rule:
- A model that fits entirely in GPU VRAM beats a larger "better" model that spills to CPU. On a Mac with a small discrete GPU (e.g. 4GB Radeon Pro), a 3B Q4 (
2GB) runs fully in VRAM via Metal at ~15-20 tok/s, while a 7B Q4 (4.7GB) spills past VRAM onto the CPU cores and drops to ~5-8 tok/s. Bigger ≠ faster when it overflows VRAM.
- Match the workload, not the leaderboard. Background/feed summarization tolerates a smaller, faster model; interactive chat wants the quality bump. Pick the smallest model that's adequate for the task.
- On a RAM-capped box (16GB), a free cloud API often beats local. If the only LLM use is non-interactive summarization, a free-tier API (Groq ~14k req/day, OpenRouter) is faster and costs zero RAM. Local is the right call only for offline/air-gapped needs.
Rough GGUF Q4_K_M sizing: 3B≈2GB, 7-8B≈4.7-5GB, 13B≈7-8GB. Keep the model under VRAM for Metal speed; keep it under ~half of total RAM so the OS + app stay healthy.
Pitfalls
- Wrong runtime-targets route. The controller exposes
GET /runtime/targets, not /api/runtime-targets. Cache TTL is 300s for targets.
- Managed llama.cpp build timeout. The controller can build llama.cpp from source, but on macOS the Homebrew bottle is much faster and avoids needing
cmake/git from scratch. Use brew install llama.cpp unless the user explicitly wants a source build.
- CPU-only machines must disable GPU layers. Always add
"n-gpu-layers": 0 in extra_args for CPU-only recipes. Otherwise llama.cpp may try to initialize Metal and crash or hang on Intel Macs.
- Recipe
served_model_name is what callers use. The chat completion request must match served_model_name (or the recipe id if no alias is set). The controller will map it to the canonical name.
- Shell quoting breaks JSON. When testing with
curl -d, use double quotes for the JSON and escape inner double quotes, or write the JSON to a file. Single-quoted JSON with embedded single quotes (e.g., 'Say 'hello'') will fail with Invalid JSON body.
- Frontend needs separate startup. The frontend is usually a Next.js dev server on port 3000. Start it with
npm run dev in the frontend directory, not via the controller. Verify /setup returns 200.
- Usage cache is 15s.
/usage is analytics, not real-time. Use include_controller=true to see controller request telemetry in the same payload.
- No model auto-launch. The chat proxy never launches a model. If the requested model isn't running, it returns a 503 OpenAI-shaped error.
Recipe template
See templates/llamacpp-cpu-recipe.json for a copy-paste starter. Adjust model_path, id, name, and served_model_name for the model you downloaded.
Verification script
See scripts/verify-local-studio.sh — a quick curl-based smoke test that checks health, runtime targets, models, launches a recipe, waits for readiness, and tests a chat completion. Copy and modify the recipe/model variables before running.
References
references/local-studio-route-map.md — full route summary with example responses from a working session.
references/llamacpp-cpu-recipe-notes.md — field-by-field recipe notes and common extra_args for llama.cpp.
references/model-sizing-constrained-mac.md — hardware probe commands + how to size a model to VRAM/RAM (the "fits-in-VRAM beats bigger" rule).
references/windows-gpu-moe-offloading.md — Windows CUDA build, MoE offloading with -ngl + -cmoe, VRAM budgets for 12GB cards.
references/localai-overview.md — LocalAI (mudler/LocalAI) condensed overview: composable-backend architecture, API surface, quickstart, vs Ollama. Alternative single-binary serving stack.
llm-provider-setup/references/kv-cache-memory-budgeting.md — KV cache memory formula and table of common models. Essential when sizing context length against RAM.
1---
2name: local-llm-serving
3description: "Set up, launch, and verify a self-hosted local LLM serving stack (controller + web UI + llama.cpp backend) on macOS/Linux. Covers model downloads, recipe creation, CPU-only inference, OpenAI-compatible proxy verification, and usage accounting."
4category: local-llm
5version: 1.0.0
6triggers:
7 - "local llm"
8 - "local studio"
9 - "self-hosted llm"
10 - "llama.cpp server"
11 - "run model locally"
12 - "download gguf"
13 - "launch recipe"
14 - "v1/models"
15 - "usage accounting"
16platforms: [macos, linux]
17metadata:
18 source: https://skillmd.com/skills/theheavenlyd3mon/local-llm-serving
19 hermes:
20 related_skills: [project-workspace, hermes-maintenance, docker-management]
21---
22
23# Local LLM Serving
24
25Class-level runbook for getting a self-hosted LLM serving stack running on a workstation: controller Hono API, Next.js frontend, llama.cpp backend, model download, recipe registration, launch, and end-to-end verification through the OpenAI-compatible proxy.
26
27## When to use this
28
29- The user wants to run a local LLM stack (Local Studio or similar controller + frontend + inference backend).
30- You need to go from "backend installed" to "chat completion works and usage is recorded."
31- The machine is CPU-only or GPU-less, and you need a recipe that avoids trying to use GPU.
32
33## Standard flow
34
351. **Check controller status**
36 ```bash
37 curl -s http://127.0.0.1:8080/health
38 curl -s http://127.0.0.1:8080/api/system-status
39 ```
40 If the controller isn't running, start it from the controller directory first.
41
422. **Install the inference backend**
43 - For llama.cpp on macOS, prefer Homebrew's bottled build to avoid the 45-minute managed source build:
44 ```bash
45 brew install --quiet cmake llama.cpp
46 llama-server --version
47 ```
48 - Verify the controller sees it: `GET /runtime/targets` (note: NOT `/api/runtime-targets`).
49
503. **Download a model**
51 Use the controller's download endpoint; pick a small GGUF for CPU (e.g., Qwen2.5-0.5B-Instruct Q4_K_M):
52 ```bash
53 curl -s -X POST http://127.0.0.1:8080/studio/downloads \
54 -H 'Content-Type: application/json' \
55 -d '{"model_id":"bartowski/Qwen2.5-0.5B-Instruct-GGUF","allow_patterns":["*Q4_K_M.gguf"]}'
56 ```
57 Then confirm with `GET /v1/studio/models`.
58
594. **Create a recipe**
60 Recipes are the controller's unit of model launch configuration. See `templates/llamacpp-cpu-recipe.json` for a known-good CPU-only starter. Required fields:
61 - `id`, `name`, `model_path` (absolute)
62 - `backend`: `llamacpp`
63 - `runtime`: `{ "kind": "binary", "ref": "/usr/local/bin/llama-server" }`
64 - `host`, `port`, `served_model_name`
65 - `extra_args`: `{ "n-gpu-layers": 0 }` for CPU-only machines
66
67 Send it to `POST /recipes`.
68
695. **Launch the model**
70 ```bash
71 curl -s -X POST http://127.0.0.1:8080/launch/<recipe-id>
72 curl -s "http://127.0.0.1:8080/wait-ready?timeout=120"
73 ```
74
756. **Verify the OpenAI-compatible surface**
76 ```bash
77 curl -s http://127.0.0.1:8080/v1/models
78 curl -s -X POST http://127.0.0.1:8080/v1/chat/completions \
79 -H 'Content-Type: application/json' \
80 -d '{"model":"<served_model_name>","messages":[{"role":"user","content":"hello"}],"max_tokens":50}'
81 ```
82
837. **Check usage accounting**
84 ```bash
85 curl -s http://127.0.0.1:8080/usage?include_controller=true
86 ```
87
88## Key route map
89
90| Endpoint | Purpose |
91|----------|---------|
92| `GET /health` | Controller liveness |
93| `GET /api/system-status` | Controller + inference status |
94| `GET /runtime/targets` | Available backends/runtimes |
95| `POST /studio/downloads` | Download a model from Hugging Face |
96| `GET /v1/studio/models` | Registered models |
97| `POST /recipes` | Register a launch recipe |
98| `POST /launch/:recipeId` | Launch a recipe |
99| `GET /wait-ready` | Wait for inference backend healthy |
100| `GET /v1/models` | OpenAI-compatible model list |
101| `POST /v1/chat/completions` | OpenAI-compatible chat completion |
102| `GET /usage` | Inference usage stats |
103| `GET /usage?include_controller=true` | Include controller request telemetry |
104
105## Windows GPU + MoE offloading
106
107For MoE models (Laguna-XS-2.1, Qwen3.6 MoE) on Windows with CUDA, combine
108`-ngl` (layer-level GPU offload) with `-cmoe` (MoE expert weights to CPU).
109These are independent mechanisms — `-ngl` controls shared weights, `-cmoe`
110controls expert tensors via regex. See `references/windows-gpu-moe-offloading.md`
111for the full runbook, build commands, and VRAM budget.
112
113## Choosing a model size for the machine
114
115Before recommending a model, **probe the actual hardware** — don't guess from the model name. Run the probe commands in `references/model-sizing-constrained-mac.md`, then size by this rule:
116
117- **A model that fits entirely in GPU VRAM beats a larger "better" model that spills to CPU.** On a Mac with a small discrete GPU (e.g. 4GB Radeon Pro), a 3B Q4 (~2GB) runs fully in VRAM via Metal at ~15-20 tok/s, while a 7B Q4 (~4.7GB) spills past VRAM onto the CPU cores and drops to ~5-8 tok/s. Bigger ≠ faster when it overflows VRAM.
118- **Match the workload, not the leaderboard.** Background/feed summarization tolerates a smaller, faster model; interactive chat wants the quality bump. Pick the smallest model that's adequate for the task.
119- **On a RAM-capped box (16GB), a free cloud API often beats local.** If the only LLM use is non-interactive summarization, a free-tier API (Groq ~14k req/day, OpenRouter) is faster and costs zero RAM. Local is the right call only for offline/air-gapped needs.
120
121Rough GGUF Q4_K_M sizing: 3B≈2GB, 7-8B≈4.7-5GB, 13B≈7-8GB. Keep the model under VRAM for Metal speed; keep it under ~half of total RAM so the OS + app stay healthy.
122
123## Pitfalls
124
125- **Wrong runtime-targets route.** The controller exposes `GET /runtime/targets`, not `/api/runtime-targets`. Cache TTL is 300s for targets.
126- **Managed llama.cpp build timeout.** The controller can build llama.cpp from source, but on macOS the Homebrew bottle is much faster and avoids needing `cmake`/`git` from scratch. Use `brew install llama.cpp` unless the user explicitly wants a source build.
127- **CPU-only machines must disable GPU layers.** Always add `"n-gpu-layers": 0` in `extra_args` for CPU-only recipes. Otherwise llama.cpp may try to initialize Metal and crash or hang on Intel Macs.
128- **Recipe `served_model_name` is what callers use.** The chat completion request must match `served_model_name` (or the recipe `id` if no alias is set). The controller will map it to the canonical name.
129- **Shell quoting breaks JSON.** When testing with `curl -d`, use double quotes for the JSON and escape inner double quotes, or write the JSON to a file. Single-quoted JSON with embedded single quotes (e.g., `'Say 'hello''`) will fail with `Invalid JSON body`.
130- **Frontend needs separate startup.** The frontend is usually a Next.js dev server on port 3000. Start it with `npm run dev` in the frontend directory, not via the controller. Verify `/setup` returns 200.
131- **Usage cache is 15s.** `/usage` is analytics, not real-time. Use `include_controller=true` to see controller request telemetry in the same payload.
132- **No model auto-launch.** The chat proxy never launches a model. If the requested model isn't running, it returns a 503 OpenAI-shaped error.
133
134## Recipe template
135
136See `templates/llamacpp-cpu-recipe.json` for a copy-paste starter. Adjust `model_path`, `id`, `name`, and `served_model_name` for the model you downloaded.
137
138## Verification script
139
140See `scripts/verify-local-studio.sh` — a quick curl-based smoke test that checks health, runtime targets, models, launches a recipe, waits for readiness, and tests a chat completion. Copy and modify the recipe/model variables before running.
141
142## References
143
144- `references/local-studio-route-map.md` — full route summary with example responses from a working session.
145- `references/llamacpp-cpu-recipe-notes.md` — field-by-field recipe notes and common `extra_args` for llama.cpp.
146- `references/model-sizing-constrained-mac.md` — hardware probe commands + how to size a model to VRAM/RAM (the "fits-in-VRAM beats bigger" rule).
147- `references/windows-gpu-moe-offloading.md` — Windows CUDA build, MoE offloading with `-ngl` + `-cmoe`, VRAM budgets for 12GB cards.
148- `references/localai-overview.md` — LocalAI (mudler/LocalAI) condensed overview: composable-backend architecture, API surface, quickstart, vs Ollama. Alternative single-binary serving stack.
149- `llm-provider-setup/references/kv-cache-memory-budgeting.md` — KV cache memory formula and table of common models. Essential when sizing context length against RAM.