# Llamacpp Tuner > Analyze and tune a running llama.cpp server for local coding agents: propose config (ctx, prompt cache, MTP), benchmark before/after, report deltas. Use to optimize llama-server for Claude Code or Aider. Don't use for vLLM, ollama, or non-llama.cpp. - Skill: `luongnv89/llamacpp-tuner` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds@latest add luongnv89/llamacpp-tuner` - Raw SKILL.md: https://api.skillmd.com/api/skills/luongnv89/llamacpp-tuner/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: luongnv89 (https://skillmd.com/u/luongnv89) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/luongnv89/llamacpp-tuner --- # llamacpp-tuner ## Overview Inspect a running `llama-server`, identify the bottlenecks that matter for a *coding agent* workload (long context, heavy prompt reuse, latency-sensitive tool calls), propose a tuned configuration, measure before/after, and report concrete deltas. The result is an actionable optimization with numbers behind it, not generic advice. ## When to use Use this skill when the user asks to **tune, optimize, or speed up llama.cpp / llama-server** for a local coding agent — phrases like "make my local model faster", "optimize llamacpp for Claude Code / Codex / Aider", "best llama-server config", or invokes `/llamacpp-tuner`. Do **not** use this skill for: vLLM, Ollama (other than as a memory-pressure suspect to free), text-generation-webui, or proprietary inference engines. The benchmark and the knobs are specific to `llama-server`. ## Workflow The skill runs as a single sequential workflow. Each step gates the next. ``` 0. remote pre-flight ← STOP if LLAMACPP_BASE_URL is remote ↓ 1. discover machine → 2. discover running server → 3. research model (fetch HF card) ↓ ↓ ↓ └──────────────────────────┴───────────────────────────┘ ↓ 4. propose config ↓ 5. user approval gate ← STOP if user rejects ↓ 6. benchmark BEFORE ↓ 7. apply config (kill+restart) ↓ 8. benchmark AFTER ↓ 9. final report ``` ### Step 0 — Remote pre-flight The tuner only makes sense against a *local* `llama-server` — it kills, restarts, and re-probes the process on the same host. None of that is meaningful (or safe) against a remote llama.cpp endpoint, where the local machine has no `llama-server` binary and no permission to restart someone else's server. Run the pre-flight gate before doing anything else: ```bash python3 scripts/tuner_preflight.py ``` The script reads `LLAMACPP_BASE_URL` via `claude_codex_local.core` (same loopback rules as the wizard) and behaves as follows: - **Local base URL** (loopback / `localhost`): exits `0` silently — continue to Step 1. - **Remote base URL**: prints the line below and exits `0`. **Stop the workflow** — do not proceed to Step 1. > llamacpp is configured as a remote endpoint (``). The tuner targets local `llama-server` instances only — skipping. If the script exits non-zero, treat it as an environment problem (e.g. the package is not importable) and surface the stderr to the user before deciding whether to continue. ### Step 1 — Discover machine Gather what platform you're targeting. The right answer for a GB10 unified-memory system is very different from a workstation with a discrete GPU + system RAM. Run, then read carefully: ```bash lscpu | grep -E "Model name|Architecture|^CPU\(s\)|Thread|Core|Socket|NUMA|MHz" free -h numactl -H 2>/dev/null || echo "no numactl" nvidia-smi 2>&1 | head -30 # Per-core max frequency reveals ARM big.LITTLE asymmetry for i in $(seq 0 $(($(nproc)-1))); do f=$(cat /sys/devices/system/cpu/cpu$i/cpufreq/cpuinfo_max_freq 2>/dev/null) echo "cpu$i max_khz=$f" done | sort -t= -k2 -nr ``` Record: - **Arch** (x86_64 vs aarch64). aarch64 may be big.LITTLE — note the perf-core CPU ids for `taskset`. - **GPU model** and whether memory is **unified** (GB10, Grace-Hopper, M-series Mac equivalents) or **discrete**. - **Free RAM and free GPU memory** *right now*. If unified memory and another LLM process (ollama, vLLM, ...) is hoarding it, **flag this** — it's the single biggest performance killer. - **NUMA topology**. Multi-socket x86 needs `numactl --interleave=all` or per-socket pinning. ### Step 2 — Discover the running server The current `llama-server` invocation tells you what was already chosen. Don't replace flags you don't understand. ```bash pgrep -fa llama-server # full cmdline ps -o pid,rss -p # memory footprint ss -tlnp | grep # confirm port curl -s http://:/health curl -s http://:/v1/models ``` Find the server log (usually a `/tmp/llama-server*.log` or wherever the user redirected stdout) and grep for the load-time configuration: ```bash grep -iE "CUDA0|n_threads|n_parallel|n_ctx|kv_unified|flash|cache-ram|prompt cache|GRAPHS|FP4|model loaded|listening" grep -iE "draft acceptance rate|statistics draft-mtp" | tail -20 ``` Record: - Current ctx, parallel, batch/ubatch, threads, KV cache type, FA on/off, prompt-cache size. - **Free GPU memory at server start** (vs total) — exposes memory pressure. - **MTP draft acceptance rate** if speculative decoding is on. <30% = poorly matched, >50% = excellent. ### Step 3 — Research the model (always fetch the model card) Identify the model from the loaded GGUF filename, then **always try to fetch its Hugging Face model card**. The card is the source of truth that the GGUF filename can't carry: native context length, OOM/ctx-size guidance, MoE expert counts, MTP support, and — on quantizer repos like Unsloth — the launch flags the model author actually recommends. Use this to keep Step 4 current with the model instead of relying on stale assumptions. Run the bundled fetcher. It resolves the repo from the running server's model path (the HF cache encodes it as `models--org--repo`), pulls both the structured API metadata **and** the README *prose*, extracts only the perf-relevant lines, and deliberately drops sampling params (this skill never tunes the sampler): ```bash # Preferred: let the path resolve the repo id (model path comes from Step 2's `pgrep -fa`) python3 scripts/fetch_model_card.py --model "" # Or pass the repo id explicitly (read it from /v1/models or /props if the path isn't an HF cache path) python3 scripts/fetch_model_card.py --repo / ``` The script is **best-effort by design**: a 404, a private/gated repo, or no network prints a one-line note and exits `0` — it never blocks the workflow. If it can't fetch the card, fall back to the server's own metadata and the GGUF filename, and mark Step 3 **PARTIAL** (not FAIL): ```bash curl -s http://:/props 2>/dev/null | python3 -m json.tool | head -40 ``` If the card resolves to a *quantizer* repo (Unsloth, bartowski, lmstudio-community) whose card is thin on architecture facts, the fetcher prints the upstream `base_model` repo — re-run with `--repo ` to get `n_ctx_train`, expert counts, and MTP support from the source. Record (prefer the card; the GGUF filename and the user's mental model can disagree): - **Total vs active params** (MoE: e.g. 35B-A3B = 35B total, ~3B active per token). Tells you whether decode is bandwidth-bound or compute-bound. The card's expert counts (`Number of Experts` / `Number of Activated Experts`) confirm this. - **n_ctx_train** (native max context, from the card's `context_length` / "Context Length: … natively"). Is the current `--ctx-size` leaving headroom on the table, or already at the ceiling? Note any author OOM guidance (e.g. "reduce to 32,768"). - **MTP / multi-token prediction support** — the card, GGUF metadata, or repo name will tell you. - **Quant**: Q4_K_M, Q5_K_M, UD-Q*, IQ*, Q8_0. Decode speed and memory cost scale with this. - **Author config recommendations** — any `llama-server` flags, output-length, or context advice the card publishes. Carry these into Step 4 as **candidate experiments to benchmark**, not silent overrides — see Step 4. ### Step 4 — Propose the optimized configuration Read `references/agent-tuning-knobs.md` for the full knob-by-knob rationale. Apply the priorities in this order: 1. **Free unified memory first** (if applicable). `ollama stop ` or kill idle LLM runners. On unified-memory systems, every GB held by another process is one less for the GPU compute path. 2. **`--parallel 1`** for single-user agent profile. Default auto-allocates 4 slots and fragments the prompt cache. 3. **`--cache-ram` to 32768 (or larger)**. The biggest single agent win — warm-turn TTFT drops from seconds to tens of ms because the cached prefix is reused. 4. **`--flash-attn on`** (explicit, not `auto`). 5. **`--cache-type-k q8_0 --cache-type-v q8_0`**. Halves KV memory at ~no quality cost. Required for KV q4_0 later. 6. **`--ctx-size`** sized to actual agent need. 65536 if your agent stays under 64k; 131072 for heavier loops; 262144 only if you genuinely use it (KV cost is linear). 7. **`--ubatch-size 1024–2048`** (sweep). Default 512 leaves prefill throughput on the table. Bigger ubatch helps long-prompt cold prefill the most. 8. **`--batch-size 4096`**. Logical batch ceiling; doesn't cost memory unless ubatch fills it. 9. **`--threads N` matched to perf cores only** (on big.LITTLE: `taskset -c ` and set `--threads N` = number of perf cores). On x86 SMT: physical-core count. 10. **`--spec-draft-n-max`** at 5–6 for code (higher acceptance than prose). Sweep 3/5/8 once stable. Do **not** apply on unified-memory systems: `-cmoe`, `-ncmoe` (no benefit — same memory pool). **Folding in the model card (from Step 3):** treat the card as ground truth for model *facts* (n_ctx_train, MTP support, quant, expert counts) — let those constrain your numbers (e.g. never propose `--ctx-size` above the card's native context). But treat author-recommended *flags* as **candidate experiments to benchmark**, not silent overrides. The skill's opinionated, workload-specific knobs (`--parallel 1`, `--cache-ram`, perf-core pinning) win on conflict because they encode the *coding-agent* profile the card author didn't assume — but **surface any conflict** to the user in one line rather than discarding it silently (e.g. "card suggests output length 65536; benchmark uses 256 — your agent's real output length should drive `--n-predict`"). Ignore card sampling params entirely; this skill never tunes the sampler. Output a config diff table: column 1 current flag value, column 2 proposed, column 3 one-line rationale per row. Don't bury the user in prose. ### Step 5 — User approval gate Show the config diff, the predicted impact (e.g. "warm TTFT should drop ~100× because the 32GB prompt cache will catch repeated project context"), and **ask the user to accept**. The skill applies a destructive action — stopping their server and restarting — and the user may have in-flight requests or dependencies. Phrase as a single yes/no question. Do not proceed without approval. If declined, output the proposed config as a paste-ready command and stop. ### Step 6 — Benchmark BEFORE Use `scripts/bench_agent.py` against the **currently running** server. The script runs five prompt sizes (256 / 2k / 8k / 32k / 64k) and reports both **cold** (`cache_prompt=false`) and **warm** (`cache_prompt=true`, same prompt re-sent) numbers. Warm numbers are the ones that matter for an agent. ```bash python3 scripts/bench_agent.py --base http://127.0.0.1: --out /tmp/bench-before.json ``` Save the result. Don't apply the config yet. ### Step 7 — Apply config 1. Capture the exact current `llama-server` command line (from `pgrep -fa`). 2. Stop the running server with `kill `, wait up to 20s for the port to free. 3. Launch the new config under `nohup`, redirecting stdout to a fresh log file. Use `taskset -c ` on big.LITTLE systems. 4. Poll `/health` until `{"status":"ok"}` or the process dies. If it dies, dump the last 50 lines of the log, restore the previous command line, and abort. ### Step 8 — Benchmark AFTER Run `scripts/bench_agent.py` again with `--out /tmp/bench-after.json`. Same cases as before — apples-to-apples comparison only works if the prompt shapes match exactly, which the script guarantees. ### Step 9 — Final report Read `references/report-format.md` for the report template. The report must contain: - **Machine + model summary** (one paragraph). - **Diff table** of flags changed (before → after, one-line rationale per row). - **Benchmark table** with both cold and warm columns, plus a **delta column** for each. - **Verdict line**: where the agent will feel this — usually "warm TTFT ↓ Nx" and "long-context decode ↑ N%". - **Caveats**: which tunings depend on workload assumptions (e.g. "prompt cache only helps if the agent re-sends overlapping context across turns — confirm with logs after a real session"). - **Optional next experiments** with the single-flag commands to run them. Output Step Completion Report after each major step using the format described below. ## Step Completion Report format After each numbered step, emit: ``` ◆ (step N of 9 — ) ·································································· : √ pass — : × fail — : √ pass ____________________________ Result: PASS | FAIL | PARTIAL ``` `PASS` lets the workflow continue. `FAIL` stops it — never apply a config or run a benchmark from a failed discovery state. ## Acceptance Criteria A run is complete when **all** of the following hold: - [ ] Step Completion Report emitted for each of the 9 workflow steps (10 with Step 0); none ended `FAIL`. - [ ] Step 0 ran first; if `LLAMACPP_BASE_URL` was remote the skill exited cleanly with the documented skip message and did **not** touch any of Steps 1–9. - [ ] Step 3 **attempted** a model-card fetch (`scripts/fetch_model_card.py`). This is best-effort: if the card was fetched, its facts (n_ctx_train, quant, MTP) informed Step 4 and any author flag recommendations were carried in as labelled experiments; if the fetch missed (404 / gated / offline), Step 3 was marked PARTIAL with the note and the run continued. A failed fetch is **not** grounds to FAIL the run. - [ ] `/tmp/bench-before.json` and `/tmp/bench-after.json` exist and cover the same five prompt sizes (256 / 2k / 8k / 32k / 64k), both cold and warm. - [ ] The final report contains: machine+model paragraph, before→after flag diff table, benchmark table with cold/warm/delta columns, verdict line, caveats, optional next experiments. - [ ] User approval was captured at Step 5 before any destructive action (server restart). - [ ] If the new server failed `/health`, the previous command line was restored and the run aborted — no half-applied state. - [ ] Warm TTFT delta is reported as a multiplier (e.g. `19×`), not just a percent — agents feel the multiplier, not the percent. - [ ] Any recommendation that depends on a workload assumption (KV q4_0, MTP draft n) is labelled as an *experiment*, not a default. If a criterion cannot be satisfied (e.g. server refuses to restart), the report must include a `## Blockers` section naming the criterion and the reason. ## Expected output A passing final report follows `references/report-format.md`. A trimmed example of the verdict-bearing section: ``` ◆ Final report — llamacpp-tuner ·································································· Machine: NVIDIA GB10 unified-memory, 128 GB, aarch64 perf cores 0-9 Model: Qwen3-Coder-30B-A3B UD-Q4_K_XL, n_ctx_train=262144, MTP=on Flag diff (8 changes): --parallel 4 → 1 single-agent profile --cache-ram 0 → 32768 prompt cache reuse --flash-attn auto → on force FA path --cache-type-k f16 → q8_0 halve KV mem --cache-type-v f16 → q8_0 halve KV mem --ctx-size 32768 → 131072 agent loops up to 100k --ubatch-size 512 → 2048 prefill throughput --spec-draft-n-max — → 5 MTP for code Benchmarks (cold | warm, prompt_tok/s • decode_tok/s): size BEFORE AFTER Δ warm TTFT 256 412 • 38 480 • 41 | 451 • 39 1820 • 42 19× 2048 398 • 36 460 • 38 | 442 • 37 1740 • 40 18× 8192 361 • 34 420 • 36 | 410 • 35 1600 • 38 16× 32768 240 • 28 285 • 29 | 290 • 30 1100 • 31 13× 65536 142 • 22 170 • 23 | 175 • 24 640 • 25 9× Verdict: warm TTFT ↓ 13-19× across all sizes; long-ctx decode ↑ 7-9%. Caveats: Prompt cache only pays back when the agent re-sends overlapping context. Next: Sweep --spec-draft-n-max ∈ {3,5,8} during a real coding session. ``` The exact numbers will differ per machine and model; the *shape* (flag diff → benchmark table → verdict → caveats → next) is the testable contract. ## Resources - `scripts/tuner_preflight.py` — Step 0 gate. Exits 0 with a skip message when `LLAMACPP_BASE_URL` is remote, exits 0 silently when local. Imports `_is_local_base_url` / `llamacpp_base_url` from `claude_codex_local.core`. - `scripts/fetch_model_card.py` — Step 3 model-card fetcher. Resolves the HF repo from the GGUF path (or `--repo`), pulls structured metadata + README prose, extracts perf-relevant config notes (dropping sampling params), and resolves the upstream `base_model`. Best-effort: a 404 / offline / gated repo prints a note and exits 0 without blocking. `--json` for machine-readable output. - `scripts/bench_agent.py` — coding-agent shaped benchmark. Sends five prompt sizes, both cold and warm, parses the server's `timings` field, writes machine-readable JSON. - `references/agent-tuning-knobs.md` — every flag this skill might recommend, why, and what it costs. - `references/report-format.md` — the final-report template (full version). ## Notes & gotchas - **Never apply KV `q4_0` blindly** for a coding agent. It does lift long-ctx decode 15–30% but trades fidelity that matters when the model has to recall specific code references. Recommend it only as an *experiment* with a quality spot-check. - **MTP draft acceptance is workload-dependent**. The number measured during a synthesis prompt (e.g. "summarize") is not the number you'll see on real code completions. Re-measure with a representative prompt before locking `--spec-draft-n-max`. - **The benchmark prompt deliberately avoids early EOS** by asking for a long structured review. If you change the prompt template, verify `n_predict` is actually reached on at least the medium cases — otherwise decode tok/s is reliable but wall-clock comparisons are not. - **Unified memory systems** (GB10, Grace-Hopper, Apple Silicon equivalents): MoE CPU offload (`-cmoe`, `-ncmoe`) is pointless — CPU and GPU share the same memory. - **The skill is read-only on the model file**. It never re-quantizes, re-converts, or modifies the GGUF. Restart-only configuration changes.