Local Model Serving

Local Model Serving

Use this skill whenever a task needs local LLM inference on this machine. It is engine- and model-generic: the measured numbers come from a Qwen3.8-27B-class dense model (8-bit, ~28-30 GB resident) on a 128 GB Apple Silicon Mac, but the decision rules and settings transfer to any model served by the same two engine families.

The Two Engine Families (same model, two backends)

Both are OpenAI-compatible (/v1/chat/completions), so the same client code works against either — only the base URL and the thinking-off parameter differ.

Drafters (DSpark, DFlash 2, and similar speculative-decoding heads) are modes of the MLX family, not a third engine. Do not install NVIDIA-only stacks (SGLang, vLLM) or a third MLX GUI to A/B a drafter. A better drafter can raise short-context decode; it does not remove the long-context crossover, because every verify round still walks the full KV. Use the drafter the local serving policy or helper names. llama.cpp may be LM Studio or llama-server; that is still the same family.

First: The Memory Rule (crash prevention)

Never run both engines with a model resident at the same time. On a 128 GB machine, two resident 28 GB models plus the OS drove the system into ~20-46 GB of swap and crushed both engines' throughput (a long run was aborted, another finished at half speed). The same exclusivity rule applies on a 64 GB machine that can fit only one ~28 GB model. Each engine fits alone; both do not.

• Stop the other engine before starting one: lms unload <identifier> / lms unload all, or pkill -f "mlx-dspark serve" (or the equivalent for your server).
• Wrap start/stop in a small helper that refuses the second engine unless it unloads the first — mechanical enforcement beats discipline.

Decision Rule

Rule of thumb: short in, long out → MLX spec-decode; long in → llama.cpp.

Settings (pinned, per engine)

• Context: use the model's native window (e.g. 262,144) unless the task explicitly needs less.
• Thinking: pin it off unless the task is reasoning-specific. Default thinking produces reasoning-only/truncated output in structured pipelines. The knob name differs per engine — using the wrong one silently does nothing, so verify with a smoke call.
• Reproducible runs (evals): temperature 0 (greedy), and compare runs only within one engine + quant — Q8_0 vs int8 and sampler differences shift scores.
• Realistic generation (Qwen3.8 documented defaults): temp 0.7, top_p 0.80, top_k 20, presence_penalty 1.5.

Known Workload Mapping

Verify The Engine Is Actually Serving

Smoke-test before a batch run — confirm reachability and that thinking is off (the wrong knob is the classic silent failure):

curl -s http://127.0.0.1:<port>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Say OK"}],
       "max_tokens":8,"<thinking-off-knob>":<value>}'

Closeout

• Unload when done (lms unload / stop the server process) so memory returns for the other engine or other workloads on the same machine.
• If a run produced a new measurement or decision (quality delta, throughput, engine crossover), record it in your repository's knowledge base with the exact setup so it is reproducible — single-run anecdotes are signal, not proof.