Optimize Local Model Compression

Optimize Local Model Compression

Compression (quantization to 4-bit) is mandatory for local deployment — it is the difference between a 9.5 GB model and a 3.6 GB model, between 23 tok/s and 45 tok/s. But standard compression methods optimize for general text quality (perplexity, MMLU) and silently destroy tool-call fidelity. This skill teaches the method we invented to fix that: outcome-optimized, layer-aware compression that protects the circuits responsible for structured output.

This is not theory. Every recommendation below is backed by measured results on Gemma 4 E2B through 31B, on a 103-row tool-call benchmark and a 14-task multi-turn agent board, on Apple Silicon (M5 Max, 128 GB), at zero API cost.

When to use

• The user's compressed model emits broken JSON, fails to call tools, or scores lower on tool-calling tasks than the hosted version.
• The user is choosing between quantization methods (naive 4-bit, QAT, OptiQ, or a custom conversion).
• The user wants to convert a model and is asking which settings matter.
• A local eval shows a model "can't do tool calls" but the hosted version can.

Safety Gates

• No conversion without explicit approval. Conversions are compute-heavy (15-30 min GPU time) and produce large artifacts. State the target model, BPW, and expected output size first.
• Background long conversions. Sensitivity probing takes 15-30 min; background it and keep working.
• Do not run conversions during meetings. GPU-bound work can freeze external monitors on Apple Silicon. Check for active display connections before starting.

The three things that matter

1. Group size must match the training noise profile

If the model is a QAT checkpoint (Google's q4_0 format), the downstream quantization group_size must match the QAT block structure (32). The default in most frameworks (MLX: 64, llama.cpp: varies) silently breaks QAT's training-time hardening.

At E4B scale the damage is worse: g64 scores 0.155 (64 errors) vs g32 at 0.252 (37 errors) — a 10pp swing from one parameter.

How to check: read the model's config.json for quantization_config or the QAT format spec. Google's Q4_0 uses block-32. Set --group-size 32 on every MLX conversion of a QAT checkpoint.

2. Calibration data determines which layers are protected

Sensitivity-driven compression (OptiQ, AWQ) measures per-layer vulnerability by pushing calibration data through the model. What you calibrate on is what you protect. The industry default (OptiQ's 6-domain mix) uses only 17% tool-calling samples — it protects reasoning circuits, not tool-call circuits.

Using a calibration mix with 58% tool-calling traces changes the layer allocation: 113 of 276 weight tensors receive a different bit-width assignment. The layers that "wake up" as sensitive under tool-call data are concentrated in early layers (1-8) — these control the tool-call trigger circuit.

Same base model, same group size, same target BPW (5.0). Only variable is calibration data. +4.8pp from calibration alone.

3. Stacking all three optimizations beats any single method

QAT hardening, sensitivity allocation, and tool-call calibration are complementary, not alternative. Stacking all three produces the best artifact:

The stacked method recovers 98% of uncompressed BF16 fidelity at less than half the size, and is 2× faster than every other variant including BF16.

How compression breaks tool calls (the error taxonomy)

Three distinct failure modes, each with a different root cause:

Trigger failure (model emits prose instead of a tool call)

The dominant failure mode. The model writes natural-language reasoning ("I will now update the CRM record...") instead of emitting a tool-call token. This is not garbled JSON — the model understands the task but fails to switch into tool-emission mode.

• Naive 4-bit: 72.8% of rows fail this way (catastrophic)
• QAT BF16: 7.8% (QAT training specifically protects this circuit)
• Vanilla BF16: 16.5% (the base rate)

Compression degrades the gating mechanism that routes the model into tool-call format. QAT hardening protects it; naive 4-bit destroys it.

Parse failure (model attempts tool call but output is unparseable)

The model tries to emit a tool call but the structured output is malformed. Rate scales with compression aggressiveness:

• BF16: 2.9-4.9%
• Stacked (ours): 10.7%
• OptiQ default: 11.7%
• QAT g64 (broken): 16.5%

Wrong tool (model calls the wrong function)

The model emits a valid tool call but picks the wrong tool (e.g., api_fetch instead of api_search). This is roughly constant across all compression levels including BF16 (~30-47%). This is a model capability ceiling, not a compression artifact. No compression optimization can fix it.

Flow

1. Diagnose the failure mode. Is the model failing to call tools (trigger failure), emitting broken JSON (parse failure), or calling the wrong tool (capability ceiling)? Each has a different fix:
   • Trigger failure → use QAT base + tool-heavy calibration
   • Parse failure → use lower BPW target (5.0, not 4.0) + tool-heavy calibration
   • Wrong tool → upgrade to a larger model; compression can't fix this
2. Check the group size. If using a QAT checkpoint, verify group_size=32 in the conversion. See reference.md for the per-model conversion commands.
3. Choose the conversion method based on model size and workload:
   • E2B (onboarding): Stacked (QAT + g32 + tool calibration). Best in class.
   • E4B-12B (mid-range): OptiQ pre-built or stacked conversion.
   • 26B+ (workstation): OptiQ or naive; the NTC metric is less reliable here.
4. Convert and certify. Run the conversion, serve it, and verify tool-call fidelity with a scored eval. See reference.md for the exact commands and the certification checklist.
5. Record the artifact. The compressed model should be named with its method (-stacked-understudy, -qat-g32-understudy, etc.) and logged with its BPW, group size, calibration mix, and measured fidelity.

Conversion commands

See reference.md for the exact optiq convert and mlx_vlm convert commands, the calibration data format, the BPW recommendations per model size, and the certification checklist.

Output Standard

End with: the diagnosed failure mode; the recommended conversion method and why; the conversion command (stated, not yet run without approval); the expected fidelity vs the current artifact; and any approval still needed (model download, compute time, disk space).

References

• reference.md — conversion commands, calibration data format, BPW table, certification checklist, and the full experimental evidence table.
• references/browser-webgpu-inference.md — browser-based local inference via WebGPU (the Gemma 4 WebGPU Kernels reference), and how to reimplement it with Understudy's stacked QAT compression for SaaS apps that want client-side summarization, offline tool calling, and privacy-preserving inference.
• ../manage-local-models/reference.md — the verified MLX ladder, serving manifests, and quantization primer.
• ../run-local-model-lab/SKILL.md — how to score a compressed model against a real workload.