Serve Cache QA

Serve Cache QA

Use this skill when changing or debugging @mlxts/serve prompt-prefix caching, continuous scheduling, Pi/OpenAI-compatible agent loops, or Qwen/Gemma serving regressions.

Core Model

Prompt-prefix cache hits are completed retained snapshots. Cold concurrent requests cannot reuse an in-flight snapshot. Exact-boundary caches such as Qwen hybrid and Gemma layer-pattern caches can reuse only retained exact prompt boundaries unless the family snapshot explicitly supports shorter forks.

Serve owns matching, retention, eviction, event accounting, and protocol usage. Transformers owns cache state shape, layerKinds, snapshot/fork validity, and disposal.

Debug Ladder

1. Read packages/serve/AGENTS.md, docs/runtime-safety.md, and the serving entries in MEMORY.md.
2. Capture route and cache events before changing code:
   • [route] ... route=continuous|single ... model_type=...
   • [cache] ... miss|hit|write ... read_tokens=... write_tokens=...
   • generation_scheduler_phase queued, prefill, admitted, first-token, and finished phases.
   • regression:agent-cache JSON requests[] entries with client duration/TTFT/stream/cache usage and server route/cache/prefill summaries.
3. Separate cold-concurrent misses from warm-retention failures:
   • cold A+B simultaneous: misses are expected when no snapshot exists yet
   • warm exact A repeat: must hit
   • warm A/B/A with divergent sessions: must hit when retention keeps both exact prompt boundaries
4. Check protocol shape. Prompt-prefix caching applies to message/content prompts with compatible prompt-cache identity; plain text completions are not the Pi chat-loop proof path.
5. Check cache semantics through CacheLayerKind, not family names:
   • full KV can usually trim/prefix-fork
   • Gemma layer-pattern full+sliding is exact-boundary unless the snapshot says otherwise
   • Qwen hybrid full+linear-recurrent is exact-boundary unless the family-owned cache says otherwise

Required Coverage

For a cache remediation, add the narrowest unit test that reproduces the product failure without real checkpoints, then run the focused package tests.

Minimum unit coverage:

• exact-boundary A/B/A retention with Gemma-style ["full", "sliding"] CacheLayerKind
• Qwen-style exact continuation with ["linear-recurrent", "full"]
• routing tests that prove cache-shape decisions use CacheLayerKind
• multi-model router evidence that model engines keep separate cache state

Real checkpoint proof, when cached models and runtime budget are available:

• dense Gemma and Gemma MoE
• dense Qwen and Qwen MoE
• one Gemma model and one Qwen model served by the same process
• repeated Pi-style chat turn for one session, then two divergent sessions, then exact A/B/A replay

Use bun run regression:agent-cache -- --scenarios qwen-dense,gemma-dense,multi-dense for the automated dense proof. Add --include-moe when the Qwen/Gemma MoE checkpoints and memory budget are available.

For reports involving two active Gemma MoE agents, escalate to bun run regression:agent-cache -- --scenarios gemma-moe --max-concurrent-requests 2 --report-json .tmp/agent-cache-regression/gemma-moe-concurrent.json. The cold A/B requests may both miss. The warm A/B replay and exact A replay must hit retained prompt boundaries.

Use cmux for Pi/server smokes so server logs and two client terminals remain visible. Keep the in-process regression as the canonical server gate, and use the cmux smoke to explain client-shape differences such as tool schemas, thinking flags, or changed workspace prompt text. Heavy MLX commands remain exclusive.

Evidence

Runtime-sensitive cache changes need a docs/reviews/ artifact with:

• ## Files Reviewed naming every changed runtime-sensitive file
• what was reproduced or ruled out
• focused tests and real checkpoint/CMux evidence
• memory or performance tradeoff, especially retained snapshot count/bytes
• independent review result
• remaining risks and explicit out-of-scope work

Do not claim shared AGENTS-prefix reuse for exact-boundary caches unless there is an exact retained snapshot at that boundary or a family-owned cache backend that supports that fork.