Mlx Serving

Apple Silicon LLM Serving (MLX / oMLX)

Operational guidance for running and tuning local LLM inference on Apple Silicon (M-series). Covers two common backends: Apple's mlx-lm and the oMLX server built on top of MLX.

Both expose OpenAI-compatible HTTP APIs, so client code is portable. The differences are in features, packaging, and the failure modes you hit when pushing models near the limits of unified memory.

Pre-Flight: Verify Environment

Before debugging any serving issue, confirm the basics:

# Hardware: confirm Apple Silicon and unified-memory size
system_profiler SPHardwareDataType | grep -E "Model|Chip|Memory"

# Activate the venv where mlx-lm is installed before the next two probes,
# otherwise they hit system Python and may report nothing useful.
python -c "import mlx.core as mx; print(mx.__version__, mx.default_device())"
python -c "import mlx_lm; print(mlx_lm.__version__)"

# oMLX (DMG install): inspect available subcommands and run health checks
omlx --help
omlx diagnose menubar   # checks Tahoe ControlCenter visibility (oMLX-specific)

# Confirm a process is actually listening
lsof -nP -iTCP -sTCP:LISTEN | grep -E "omlx|mlx_lm"

Verify which backend is bound before debugging: both can serve OpenAI- compatible endpoints on user-chosen ports, and a client request hitting the wrong backend silently produces correct-looking but unrelated output.

Backend Decision: mlx-lm vs oMLX

Skill bias: the depth here leans oMLX — feature flags, cache controls, upstream bug patterns. mlx-lm coverage is lighter and focuses on the shared substrate.

Pick mlx-lm when: you control the Python process, you want minimal surface area, you serve one model at a time, and the model fits comfortably in RAM.

Pick oMLX when: you need a daemon that survives terminal sessions, you serve multiple models behind one endpoint, you want feature flags (turboquant, dflash, MTP) without writing them yourself, or your model exceeds RAM and you need OptiQ proxy.

Both backends share the same fundamental constraints (unified memory, KV cache pressure, model architecture quirks), so most troubleshooting in this skill applies to either.

Symptom → Cause Triage

For deeper triage on each symptom: references/symptoms.md.

Core Principles

1. Lower --max-concurrent-requests to 1 for batch-fragile architectures. oMLX defaults to --max-concurrent-requests 8. Higher concurrency exposes architecture-specific batch handling bugs (notably in cache implementations like ChunkedKVCache used by Llama-4). For affected architectures, pin concurrency to 1 at launch and bench upward only after confirming clean output at batch>1.

2. One flag at a time. Feature flags interact (e.g., dflash + turboquant_kv both touch the KV cache path). Changing two flags simultaneously and observing a regression makes attribution impossible. Bench between each change.

3. Bench, don't guess. "It feels faster" is not a tuning signal. Maintain a small bench suite (chat / coding / tool-calling correctness + throughput) and re-run it after every config change. See references/bench-methodology.md.

4. Match chat templates to model cards. Tool-calling and reasoning behavior depend on the chat template applied at request time. The model's HuggingFace card is authoritative.

5. Quantization is a curve, not a switch. turboquant_kv_bits=4.0 keeps most quality; lower bits trade quality for memory. Always include a quality cell in the bench suite — a config that passes throughput but fails coding can be worse than slower-but-correct.

6. OptiQ proxy is for the "model > RAM" case only. It builds a sensitivity-driven proxy of the model so per-layer quant decisions can be made without holding the full model in memory. It's not a general speedup; for in-RAM models it adds startup cost without runtime benefit.

7. Server-side bugs masquerade as config problems. When a symptom reproduces across multiple config combinations, suspect upstream rather than chasing more flags. See references/upstream-bug-patterns.md.

References

• references/symptoms.md — symptom triage with diagnostics and fixes (OOM, batch>1 crashes, tool-call-as-text, throughput regression, log triage)
• references/omlx-feature-flags.md — per-flag reference (turboquant_kv, dflash, MTP, specprefill, thinking budget, max-concurrent-requests, force_sampling) with interactions
• references/bench-methodology.md — bench-it-don't-guess: suite design, backend-agnostic harness shape, scoring, change attribution
• references/upstream-bug-patterns.md — when to suspect upstream vs config; two recurring bug patterns (cache-shape mismatch under batched scheduling; model tool-call format not parsed) with diagnostics and reporting guidance

What This Skill Does Not Cover

• Cloud LLM hosting (Bedrock, OpenAI API, Anthropic API) — different surface entirely
• Non-MLX local backends (llama.cpp, Ollama, vLLM) — overlapping problem space but different tooling
• Model training, fine-tuning, or LoRA — see MLX training tutorials
• General Python / HTTP debugging unrelated to MLX
• Model card creation or HuggingFace upload workflows