Llamacpp Tuner

llamacpp-tuner

Overview

Inspect a running llama-server, identify the bottlenecks that matter for a coding agent workload (long context, heavy prompt reuse, latency-sensitive tool calls), propose a tuned configuration, measure before/after, and report concrete deltas. The result is an actionable optimization with numbers behind it, not generic advice.

When to use

Use this skill when the user asks to tune, optimize, or speed up llama.cpp / llama-server for a local coding agent — phrases like "make my local model faster", "optimize llamacpp for Claude Code / Codex / Aider", "best llama-server config", or invokes /llamacpp-tuner.

Do not use this skill for: vLLM, Ollama (other than as a memory-pressure suspect to free), text-generation-webui, or proprietary inference engines. The benchmark and the knobs are specific to llama-server.

Workflow

The skill runs as a single sequential workflow. Each step gates the next.

0. remote pre-flight ← STOP if LLAMACPP_BASE_URL is remote
                                   ↓
1. discover machine    →  2. discover running server  →  3. research model (fetch HF card)
        ↓                          ↓                           ↓
        └──────────────────────────┴───────────────────────────┘
                                   ↓
                          4. propose config
                                   ↓
                          5. user approval gate ← STOP if user rejects
                                   ↓
                          6. benchmark BEFORE
                                   ↓
                          7. apply config (kill+restart)
                                   ↓
                          8. benchmark AFTER
                                   ↓
                          9. final report

Step 0 — Remote pre-flight

The tuner only makes sense against a local llama-server — it kills, restarts, and re-probes the process on the same host. None of that is meaningful (or safe) against a remote llama.cpp endpoint, where the local machine has no llama-server binary and no permission to restart someone else's server.

Run the pre-flight gate before doing anything else:

python3 scripts/tuner_preflight.py

The script reads LLAMACPP_BASE_URL via claude_codex_local.core (same loopback rules as the wizard) and behaves as follows:

• Local base URL (loopback / localhost): exits 0 silently — continue to Step 1.

• Remote base URL: prints the line below and exits 0. Stop the workflow — do not proceed to Step 1.

  llamacpp is configured as a remote endpoint (<url>). The tuner targets local llama-server instances only — skipping.

If the script exits non-zero, treat it as an environment problem (e.g. the package is not importable) and surface the stderr to the user before deciding whether to continue.

Step 1 — Discover machine

Gather what platform you're targeting. The right answer for a GB10 unified-memory system is very different from a workstation with a discrete GPU + system RAM.

Run, then read carefully:

lscpu | grep -E "Model name|Architecture|^CPU\(s\)|Thread|Core|Socket|NUMA|MHz"
free -h
numactl -H 2>/dev/null || echo "no numactl"
nvidia-smi 2>&1 | head -30
# Per-core max frequency reveals ARM big.LITTLE asymmetry
for i in $(seq 0 $(($(nproc)-1))); do
  f=$(cat /sys/devices/system/cpu/cpu$i/cpufreq/cpuinfo_max_freq 2>/dev/null)
  echo "cpu$i max_khz=$f"
done | sort -t= -k2 -nr

Record:

• Arch (x86_64 vs aarch64). aarch64 may be big.LITTLE — note the perf-core CPU ids for taskset.
• GPU model and whether memory is unified (GB10, Grace-Hopper, M-series Mac equivalents) or discrete.
• Free RAM and free GPU memory right now. If unified memory and another LLM process (ollama, vLLM, ...) is hoarding it, flag this — it's the single biggest performance killer.
• NUMA topology. Multi-socket x86 needs numactl --interleave=all or per-socket pinning.

Step 2 — Discover the running server

The current llama-server invocation tells you what was already chosen. Don't replace flags you don't understand.

pgrep -fa llama-server         # full cmdline
ps -o pid,rss -p <pid>         # memory footprint
ss -tlnp | grep <port>         # confirm port
curl -s http://<host>:<port>/health
curl -s http://<host>:<port>/v1/models

Find the server log (usually a /tmp/llama-server*.log or wherever the user redirected stdout) and grep for the load-time configuration:

grep -iE "CUDA0|n_threads|n_parallel|n_ctx|kv_unified|flash|cache-ram|prompt cache|GRAPHS|FP4|model loaded|listening" <log>
grep -iE "draft acceptance rate|statistics draft-mtp" <log> | tail -20

Record:

• Current ctx, parallel, batch/ubatch, threads, KV cache type, FA on/off, prompt-cache size.
• Free GPU memory at server start (vs total) — exposes memory pressure.
• MTP draft acceptance rate if speculative decoding is on. <30% = poorly matched, >50% = excellent.

Step 3 — Research the model (always fetch the model card)

Identify the model from the loaded GGUF filename, then always try to fetch its Hugging Face model card. The card is the source of truth that the GGUF filename can't carry: native context length, OOM/ctx-size guidance, MoE expert counts, MTP support, and — on quantizer repos like Unsloth — the launch flags the model author actually recommends. Use this to keep Step 4 current with the model instead of relying on stale assumptions.

Run the bundled fetcher. It resolves the repo from the running server's model path (the HF cache encodes it as models--org--repo), pulls both the structured API metadata and the README prose, extracts only the perf-relevant lines, and deliberately drops sampling params (this skill never tunes the sampler):

# Preferred: let the path resolve the repo id (model path comes from Step 2's `pgrep -fa`)
python3 scripts/fetch_model_card.py --model "<path-to-loaded-.gguf>"

# Or pass the repo id explicitly (read it from /v1/models or /props if the path isn't an HF cache path)
python3 scripts/fetch_model_card.py --repo <org>/<repo>

The script is best-effort by design: a 404, a private/gated repo, or no network prints a one-line note and exits 0 — it never blocks the workflow. If it can't fetch the card, fall back to the server's own metadata and the GGUF filename, and mark Step 3 PARTIAL (not FAIL):

curl -s http://<host>:<port>/props 2>/dev/null | python3 -m json.tool | head -40

If the card resolves to a quantizer repo (Unsloth, bartowski, lmstudio-community) whose card is thin on architecture facts, the fetcher prints the upstream base_model repo — re-run with --repo <base_model> to get n_ctx_train, expert counts, and MTP support from the source.

Record (prefer the card; the GGUF filename and the user's mental model can disagree):

• Total vs active params (MoE: e.g. 35B-A3B = 35B total, ~3B active per token). Tells you whether decode is bandwidth-bound or compute-bound. The card's expert counts (Number of Experts / Number of Activated Experts) confirm this.
• n_ctx_train (native max context, from the card's context_length / "Context Length: … natively"). Is the current --ctx-size leaving headroom on the table, or already at the ceiling? Note any author OOM guidance (e.g. "reduce to 32,768").
• MTP / multi-token prediction support — the card, GGUF metadata, or repo name will tell you.
• Quant: Q4_K_M, Q5_K_M, UD-Q*, IQ*, Q8_0. Decode speed and memory cost scale with this.
• Author config recommendations — any llama-server flags, output-length, or context advice the card publishes. Carry these into Step 4 as candidate experiments to benchmark, not silent overrides — see Step 4.

Step 4 — Propose the optimized configuration

Read references/agent-tuning-knobs.md for the full knob-by-knob rationale. Apply the priorities in this order:

1. Free unified memory first (if applicable). ollama stop <model> or kill idle LLM runners. On unified-memory systems, every GB held by another process is one less for the GPU compute path.
2. --parallel 1 for single-user agent profile. Default auto-allocates 4 slots and fragments the prompt cache.
3. --cache-ram to 32768 (or larger). The biggest single agent win — warm-turn TTFT drops from seconds to tens of ms because the cached prefix is reused.
4. --flash-attn on (explicit, not auto).
5. --cache-type-k q8_0 --cache-type-v q8_0. Halves KV memory at ~no quality cost. Required for KV q4_0 later.
6. --ctx-size sized to actual agent need. 65536 if your agent stays under 64k; 131072 for heavier loops; 262144 only if you genuinely use it (KV cost is linear).
7. --ubatch-size 1024–2048 (sweep). Default 512 leaves prefill throughput on the table. Bigger ubatch helps long-prompt cold prefill the most.
8. --batch-size 4096. Logical batch ceiling; doesn't cost memory unless ubatch fills it.
9. --threads N matched to perf cores only (on big.LITTLE: taskset -c <perf-core-ids> and set --threads N = number of perf cores). On x86 SMT: physical-core count.
10. --spec-draft-n-max at 5–6 for code (higher acceptance than prose). Sweep 3/5/8 once stable.

Do not apply on unified-memory systems: -cmoe, -ncmoe (no benefit — same memory pool).

Folding in the model card (from Step 3): treat the card as ground truth for model facts (n_ctx_train, MTP support, quant, expert counts) — let those constrain your numbers (e.g. never propose --ctx-size above the card's native context). But treat author-recommended flags as candidate experiments to benchmark, not silent overrides. The skill's opinionated, workload-specific knobs (--parallel 1, --cache-ram, perf-core pinning) win on conflict because they encode the coding-agent profile the card author didn't assume — but surface any conflict to the user in one line rather than discarding it silently (e.g. "card suggests output length 65536; benchmark uses 256 — your agent's real output length should drive --n-predict"). Ignore card sampling params entirely; this skill never tunes the sampler.

Output a config diff table: column 1 current flag value, column 2 proposed, column 3 one-line rationale per row. Don't bury the user in prose.

Step 5 — User approval gate

Show the config diff, the predicted impact (e.g. "warm TTFT should drop ~100× because the 32GB prompt cache will catch repeated project context"), and ask the user to accept. The skill applies a destructive action — stopping their server and restarting — and the user may have in-flight requests or dependencies.

Phrase as a single yes/no question. Do not proceed without approval.

If declined, output the proposed config as a paste-ready command and stop.

Step 6 — Benchmark BEFORE

Use scripts/bench_agent.py against the currently running server. The script runs five prompt sizes (256 / 2k / 8k / 32k / 64k) and reports both cold (cache_prompt=false) and warm (cache_prompt=true, same prompt re-sent) numbers. Warm numbers are the ones that matter for an agent.

python3 scripts/bench_agent.py --base http://127.0.0.1:<port> --out /tmp/bench-before.json

Save the result. Don't apply the config yet.

Step 7 — Apply config

1. Capture the exact current llama-server command line (from pgrep -fa).
2. Stop the running server with kill <pid>, wait up to 20s for the port to free.
3. Launch the new config under nohup, redirecting stdout to a fresh log file. Use taskset -c <perf-core-ids> on big.LITTLE systems.
4. Poll /health until {"status":"ok"} or the process dies. If it dies, dump the last 50 lines of the log, restore the previous command line, and abort.

Step 8 — Benchmark AFTER

Run scripts/bench_agent.py again with --out /tmp/bench-after.json. Same cases as before — apples-to-apples comparison only works if the prompt shapes match exactly, which the script guarantees.

Step 9 — Final report

Read references/report-format.md for the report template. The report must contain:

• Machine + model summary (one paragraph).
• Diff table of flags changed (before → after, one-line rationale per row).
• Benchmark table with both cold and warm columns, plus a delta column for each.
• Verdict line: where the agent will feel this — usually "warm TTFT ↓ Nx" and "long-context decode ↑ N%".
• Caveats: which tunings depend on workload assumptions (e.g. "prompt cache only helps if the agent re-sends overlapping context across turns — confirm with logs after a real session").
• Optional next experiments with the single-flag commands to run them.

Output Step Completion Report after each major step using the format described below.

Step Completion Report format

After each numbered step, emit:

◆ <step name> (step N of 9 — <one-line context>)
··································································
  <Check 1>:          √ pass — <one-line detail>
  <Check 2>:          × fail — <reason>
  <Check 3>:          √ pass
  ____________________________
  Result:             PASS | FAIL | PARTIAL

PASS lets the workflow continue. FAIL stops it — never apply a config or run a benchmark from a failed discovery state.

Acceptance Criteria

A run is complete when all of the following hold:

• Step Completion Report emitted for each of the 9 workflow steps (10 with Step 0); none ended FAIL.
• Step 0 ran first; if LLAMACPP_BASE_URL was remote the skill exited cleanly with the documented skip message and did not touch any of Steps 1–9.
• Step 3 attempted a model-card fetch (scripts/fetch_model_card.py). This is best-effort: if the card was fetched, its facts (n_ctx_train, quant, MTP) informed Step 4 and any author flag recommendations were carried in as labelled experiments; if the fetch missed (404 / gated / offline), Step 3 was marked PARTIAL with the note and the run continued. A failed fetch is not grounds to FAIL the run.
• /tmp/bench-before.json and /tmp/bench-after.json exist and cover the same five prompt sizes (256 / 2k / 8k / 32k / 64k), both cold and warm.
• The final report contains: machine+model paragraph, before→after flag diff table, benchmark table with cold/warm/delta columns, verdict line, caveats, optional next experiments.
• User approval was captured at Step 5 before any destructive action (server restart).
• If the new server failed /health, the previous command line was restored and the run aborted — no half-applied state.
• Warm TTFT delta is reported as a multiplier (e.g. 19×), not just a percent — agents feel the multiplier, not the percent.
• Any recommendation that depends on a workload assumption (KV q4_0, MTP draft n) is labelled as an experiment, not a default.

If a criterion cannot be satisfied (e.g. server refuses to restart), the report must include a ## Blockers section naming the criterion and the reason.

Expected output

A passing final report follows references/report-format.md. A trimmed example of the verdict-bearing section:

◆ Final report — llamacpp-tuner
··································································
Machine:  NVIDIA GB10 unified-memory, 128 GB, aarch64 perf cores 0-9
Model:    Qwen3-Coder-30B-A3B UD-Q4_K_XL, n_ctx_train=262144, MTP=on

Flag diff (8 changes):
  --parallel              4        → 1        single-agent profile
  --cache-ram             0        → 32768    prompt cache reuse
  --flash-attn            auto     → on       force FA path
  --cache-type-k          f16      → q8_0     halve KV mem
  --cache-type-v          f16      → q8_0     halve KV mem
  --ctx-size              32768    → 131072   agent loops up to 100k
  --ubatch-size           512      → 2048     prefill throughput
  --spec-draft-n-max      —        → 5        MTP for code

Benchmarks (cold | warm, prompt_tok/s • decode_tok/s):
  size    BEFORE              AFTER               Δ warm TTFT
  256     412 • 38   480 • 41 | 451 • 39   1820 • 42   19×
  2048    398 • 36   460 • 38 | 442 • 37   1740 • 40   18×
  8192    361 • 34   420 • 36 | 410 • 35   1600 • 38   16×
  32768   240 • 28   285 • 29 | 290 • 30   1100 • 31   13×
  65536   142 • 22   170 • 23 |  175 • 24    640 • 25    9×

Verdict:    warm TTFT ↓ 13-19× across all sizes; long-ctx decode ↑ 7-9%.
Caveats:    Prompt cache only pays back when the agent re-sends overlapping context.
Next:       Sweep --spec-draft-n-max ∈ {3,5,8} during a real coding session.

The exact numbers will differ per machine and model; the shape (flag diff → benchmark table → verdict → caveats → next) is the testable contract.

Resources

• scripts/tuner_preflight.py — Step 0 gate. Exits 0 with a skip message when LLAMACPP_BASE_URL is remote, exits 0 silently when local. Imports _is_local_base_url / llamacpp_base_url from claude_codex_local.core.
• scripts/fetch_model_card.py — Step 3 model-card fetcher. Resolves the HF repo from the GGUF path (or --repo), pulls structured metadata + README prose, extracts perf-relevant config notes (dropping sampling params), and resolves the upstream base_model. Best-effort: a 404 / offline / gated repo prints a note and exits 0 without blocking. --json for machine-readable output.
• scripts/bench_agent.py — coding-agent shaped benchmark. Sends five prompt sizes, both cold and warm, parses the server's timings field, writes machine-readable JSON.
• references/agent-tuning-knobs.md — every flag this skill might recommend, why, and what it costs.
• references/report-format.md — the final-report template (full version).

Notes & gotchas

• Never apply KV q4_0 blindly for a coding agent. It does lift long-ctx decode 15–30% but trades fidelity that matters when the model has to recall specific code references. Recommend it only as an experiment with a quality spot-check.
• MTP draft acceptance is workload-dependent. The number measured during a synthesis prompt (e.g. "summarize") is not the number you'll see on real code completions. Re-measure with a representative prompt before locking --spec-draft-n-max.
• The benchmark prompt deliberately avoids early EOS by asking for a long structured review. If you change the prompt template, verify n_predict is actually reached on at least the medium cases — otherwise decode tok/s is reliable but wall-clock comparisons are not.
• Unified memory systems (GB10, Grace-Hopper, Apple Silicon equivalents): MoE CPU offload (-cmoe, -ncmoe) is pointless — CPU and GPU share the same memory.
• The skill is read-only on the model file. It never re-quantizes, re-converts, or modifies the GGUF. Restart-only configuration changes.