MLX model porting and optimization
Mission
Produce or inspect a correct, reproducible, architecture-aware MLX implementation. Correctness before speed. Every speed or memory claim must name hardware, software versions, workload, baseline, and quality gate.
Six families have scaffolds (MoE/SSM synthetic, others runbook-guided); four
have worked packets under examples/: Qwen2.5,
BGE, t5-small, HuBERT. Exact output is the built-in metric.
When to use this skill
Port, convert, run, inspect, quantize, package, or publish a PyTorch/Hugging Face model or MLX project on Apple Silicon, or fix any parity, NaN/Inf, shape, tokenizer, preprocessing, output, performance, memory, cache, serving, benchmark, or provenance issue in a port. Not for CUDA/non-Apple targets, ML theory without an MLX target, or training from scratch.
Trigger map
| Signal |
Load |
Port/convert/run request, config.json, safetensors index, model directory, or Hub id. |
intake, then Workflow 1-6; worked chain for dense decoders, routed runbooks otherwise. |
| User points at an existing local MLX project, running MLX app, or completed MLX port. |
inspector mode plus inspect_mlx_project.py |
| User asks "what can I do with this model?", asks for capability fit, or wants model-specific advice. |
model advisor playbook |
| A known architecture family needs the right runbook. |
model support map, then the architecture table in Workflow step 4 below |
| Dense decoder, BERT encoder, T5 encoder-decoder, HuBERT/Wav2Vec2 acoustic encoder, sparse MoE, or selective SSM. |
Use scaffold_port.py; select capture/parity mode dense-decoder (default), encoder, encoder-decoder, asr, or ssm. |
| NaN, Inf, cosine-similarity drift, parity failure, or garbage output appears versus the source. |
failure atlas |
| Weight conversion, key mapping, tensor rename, transpose, reshape, split, merge, or shape transform is in scope. |
core porting method |
| The user says "make it faster" but no profile, workload, or baseline exists yet. |
benchmarking |
| Speedup plan, how techniques combine, or expected compound gains. |
compound stacks |
| KV cache, long context, recurrent state, attention memory, or prefill/decode memory is the bottleneck. |
attention and KV cache |
| Quantization or "4-bit". |
guide, quality gate |
| Structured local optimization sweep. |
loop |
| Decoding, serving, speculative decoding, batching, streaming, or API runtime behavior is requested. |
decoding and serving |
Compile behavior, mx.compile, custom kernel, graph capture, Metal, or operation fusion comes up. |
compile and kernels |
| Publish, release, checkpoint conversion, model card, provenance, or license packaging is requested. |
packaging and publication |
| The user asks for "50-100 optimization ideas", a deep model-specific hunt, or research-backed candidates. |
hypothesis-led learning |
| Vision-language, multimodal, or image+text (VLM) input appears. |
multimodal/omni runbook |
| Diffusion, flow-matching, or image/video generation appears. |
diffusion/flow runbook |
| Text-to-speech, vocoder, or audio generation appears. |
flow-TTS, autoregressive audio |
| ASR, transcription, or streaming speech appears. |
ASR, streaming speech |
| Sparse mixture-of-experts or top-k expert routing appears. |
MoE runbook |
| Selective state-space, Mamba, or linear-attention hybrid appears. |
SSM/hybrid runbook |
| Graph, GNN, message passing, node/edge features, or sparse graph workload appears. |
graph message passing runbook |
| Classic CV detection, segmentation, keypoints, depth, OCR, or non-generative vision appears. |
non-generative CV runbook |
| Time-series, forecasting, tabular sequence, anomaly detection, or temporal model appears. |
time-series forecasting runbook |
Re-consult the map on a new config, parity failure, performance complaint, or
publish request.
Non-negotiable rules
- Do not execute untrusted model code during intake. Inspect JSON, safetensors headers, source files, and licenses statically. Treat
auto_map, custom modules, install hooks, and trust_remote_code as review gates.
- Pin the source. Record repository, revision, model files, tokenizer/processor revision, license, and checksum or artifact manifest.
- Build a source oracle before porting. Freeze deterministic fixtures and capture intermediate tensors at meaningful boundaries.
- Port the smallest eager path first. No quantization, compilation, custom kernels, batching, or speculative decoding until basic parity passes.
- Change one optimization dimension at a time. Keep a measurement and rollback record.
- Prefer native MLX operations. Try built-in fused operations, layout changes, cache design, and
mx.compile before a custom Metal kernel.
- Do not translate CUDA folklore mechanically. A CUDA technique is only a research candidate until its Metal/MLX bottleneck and implementation are demonstrated.
- Never hide quality regressions behind throughput. For audio, language, vision, and generative models, use task-specific quality checks in addition to tensor tolerances.
- Do not publish converted weights without license and provenance checks. Preserve the original model card, attribution, generation config, tokenizer/processor files, and conversion recipe.
- Daily research automation is review-only. It may collect and rank candidates, but must not silently rewrite runbooks or merge recommendations. Use the promotion-review ledger to separate review-ready findings from validation backlog and rejected leads.
- Experimental approaches require explicit opt-in. Label unvalidated contributor, blog, paper, or repository learnings as experimental, state the missing validation gate, and ask before executing them: “This is an experimental approach. Do you want to try it?” Continue only if the user says to try it.
Workflow
1. Inspect and classify
For source models, run scripts/inspect_model.py, then scripts/recommend_optimizations.py, then scripts/make_port_plan.py --artifact-root MODEL --recommendations .... An actionable plan re-inspects those bytes and recomputes the full recommendation report before advice; a blocked inspection yields only a remediation plan, and no override or registry read bypasses its blockers. An override may reorder a hybrid route but must preserve every routed family, runbook, and trait. For existing MLX projects or running ports, run scripts/inspect_mlx_project.py and read inspector mode first.
Read intake and routing. Confirm source, risk, Mac, memory, performance, quality; record ambiguity in PORT_PLAN.md. For advice, read model advisor playbook and separate results into its five controlled advisor buckets. Numeric output may come only from assets/effective_claims.json; missing gates withhold the number.
2. Select the closest proven MLX reference
Consult model support map and assets/architectures.yaml. Prefer official MLX, Apple projects, pinned third-party MLX evidence, paper-only candidates, then new code; keep support scope explicit and verify config/layout.
3. Establish the source oracle
Run scripts/capture_oracle.py against the pinned local Hugging Face model before the MLX graph, then follow parity and testing. It records inputs, embeddings, blocks, final norm, logits, hookable branches, and greedy IDs in a bounded NPZ plus manifest.
4. Implement the minimal eager MLX graph
Read core porting method; choose via the trigger map and the family records in assets/architectures.yaml. Load every runbook a hybrid route returns; the registry is the full inventory—never substitute an abbreviated list.
Scaffold an unblocked dense decoder, sparse-MoE decoder, BERT encoder, T5 encoder-decoder, HuBERT/Wav2Vec2 acoustic encoder, or opt-in minimal_selective SSM:
python3 scripts/scaffold_port.py inspection.json --artifact-root MODEL --output mlx_port
It re-inspects and fails closed; review unsupported config, never patch around a generator blocker.
Start eager: FP, batch one unless intrinsic, no compile/kernels, state/cache, assertions, reversible map.
5. Convert weights deterministically
Draft with scripts/convert_checkpoint.py --emit-draft-map, resolve the
schema-2 WEIGHT_MAP, validate with scripts/validate_weight_map.py, then
convert. Reject shard, coverage, shape, draft, or unresolved gaps; never mask
exceptions.
6. Pass the parity ladder
After conversion, scripts/run_parity.py --mode MODE runs source/MLX capture
and stops at the first failed rung. Modes: dense-decoder (default), encoder,
encoder-decoder, ssm, asr. capture_mlx.py retains captures,
compare_tensors.py extras, _capture_common.py bounded rules.
When parity fails, use failure atlas; do not optimize a failing graph.
7. Profile before choosing optimizations
Read benchmarking. Separate prefill, decode, postprocess, compile, memory, movement, sync, Python overhead; use scripts/benchmark_command.py.
8. Apply the optimization ladder
Use guides: C,KV,S,Q,T,CS.
Tiers: assets/recommendation-taxonomy.yaml; for an experimental approach, state the gate and use the rule-11 opt-in prompt.
Post-parity: one dimension; Metal only after proven bottleneck. Record hypothesis, gates, metrics, decision.
9. Package and publish
Follow packaging and publication. Include source, conversion, versions, artifacts, quantization, smoke, benchmarks, limits, license/attribution; no unsupported “faster” wording.
10. Return an engineering report
Summarize architecture/runbook, source/evidence, implementation/weights, parity, metrics, optimizations, risks, commands, and artifacts. Use assets/.
When to stop
Stop when license, remote code, unresolved parity, regressions, missing kernel fallback/tests, or missing performance metadata blocks the result.
Maintenance
maintenance, hypothesis-led learning, research loop, evidence graph. Run scripts/audit_skill.py --strict and scripts/validate_sources.py before release.
Keep tests/test_scenarios.py at full family coverage when assets/architectures.yaml changes.
1---2name: mlx-model-porting3description: Guides and validates architecture-aware ports of PyTorch/Hugging Face models to Apple MLX, inspects existing local MLX projects, and plans evidence-gated optimizations for Apple Silicon. Use when the user asks to run, port, convert, inspect, quantize, benchmark, or fix a model (LLM, VLM, audio/TTS/ASR, diffusion, SSM, MoE) for MLX, MLX-LM, MLX-VLM, MLX-Audio, or a Mac - e.g. "port this HF model to my Mac", "inspect this MLX app", "run Qwen on Apple Silicon", "convert these safetensors to MLX", "make this faster on my M3", "fix NaN in my MLX port", "speed up prefill / KV cache / speculative decoding", "publish an MLX checkpoint". Also use mid-task when a config.json, safetensors index, weight-shape or tokenizer mismatch, or Metal kernel question appears. Do not use for CUDA-only optimization, non-Apple hardware targets, or general PyTorch/ML questions with no MLX or Apple Silicon connection.4license: Apache-2.05---67# MLX model porting and optimization89## Mission1011Produce or inspect a **correct, reproducible, architecture-aware MLX implementation**. Correctness before speed. Every speed or memory claim must name hardware, software versions, workload, baseline, and quality gate.1213Six families have scaffolds (MoE/SSM synthetic, others runbook-guided); four14have worked packets under [examples/](examples/porting-patterns.md): Qwen2.5,15BGE, t5-small, HuBERT. Exact output is the built-in metric.1617## When to use this skill1819Port, convert, run, inspect, quantize, package, or publish a PyTorch/Hugging Face model or MLX project on Apple Silicon, or fix any parity, NaN/Inf, shape, tokenizer, preprocessing, output, performance, memory, cache, serving, benchmark, or provenance issue in a port. Not for CUDA/non-Apple targets, ML theory without an MLX target, or training from scratch.2021## Trigger map2223| Signal | Load |24| --- | --- |25| Port/convert/run request, `config.json`, safetensors index, model directory, or Hub id. | [intake](references/intake-and-routing.md), then Workflow 1-6; [worked chain](examples/worked-port-qwen2.5-0.5b-instruct/README.md) for dense decoders, routed runbooks otherwise. |26| User points at an existing local MLX project, running MLX app, or completed MLX port. | [inspector mode](references/inspector-mode.md) plus [inspect_mlx_project.py](scripts/inspect_mlx_project.py) |27| User asks "what can I do with this model?", asks for capability fit, or wants model-specific advice. | [model advisor playbook](references/model-advisor-playbook.md) |28| A known architecture family needs the right runbook. | [model support map](references/model-support-map.md), then the architecture table in Workflow step 4 below |29| Dense decoder, BERT encoder, T5 encoder-decoder, HuBERT/Wav2Vec2 acoustic encoder, sparse MoE, or selective SSM. | Use `scaffold_port.py`; select capture/parity mode `dense-decoder` (default), `encoder`, `encoder-decoder`, `asr`, or `ssm`. |30| NaN, Inf, cosine-similarity drift, parity failure, or garbage output appears versus the source. | [failure atlas](references/failure-atlas.md) |31| Weight conversion, key mapping, tensor rename, transpose, reshape, split, merge, or shape transform is in scope. | [core porting method](references/porting-core.md) |32| The user says "make it faster" but no profile, workload, or baseline exists yet. | [benchmarking](references/benchmarking.md) |33| Speedup plan, how techniques combine, or expected compound gains. | [compound stacks](references/compound-stacks.md) |34| KV cache, long context, recurrent state, attention memory, or prefill/decode memory is the bottleneck. | [attention and KV cache](references/attention-and-kv.md) |35| Quantization or "4-bit". | [guide](references/quantization.md), [quality gate](references/quantization-quality-gate.md) |36| Structured local optimization sweep. | [loop](references/optimization-loop.md) |37| Decoding, serving, speculative decoding, batching, streaming, or API runtime behavior is requested. | [decoding and serving](references/decoding-and-serving.md) |38| Compile behavior, `mx.compile`, custom kernel, graph capture, Metal, or operation fusion comes up. | [compile and kernels](references/compile-and-kernels.md) |39| Publish, release, checkpoint conversion, model card, provenance, or license packaging is requested. | [packaging and publication](references/packaging-and-publication.md) |40| The user asks for "50-100 optimization ideas", a deep model-specific hunt, or research-backed candidates. | [hypothesis-led learning](references/hypothesis-led-learning.md) |41| Vision-language, multimodal, or image+text (VLM) input appears. | [multimodal/omni runbook](references/runbook-multimodal-omni.md) |42| Diffusion, flow-matching, or image/video generation appears. | [diffusion/flow runbook](references/runbook-diffusion-flow.md) |43| Text-to-speech, vocoder, or audio generation appears. | [flow-TTS](references/runbook-flow-tts.md), [autoregressive audio](references/runbook-autoregressive-audio.md) |44| ASR, transcription, or streaming speech appears. | [ASR](references/runbook-asr.md), [streaming speech](references/runbook-streaming-speech.md) |45| Sparse mixture-of-experts or top-k expert routing appears. | [MoE runbook](references/runbook-moe-transformer.md) |46| Selective state-space, Mamba, or linear-attention hybrid appears. | [SSM/hybrid runbook](references/runbook-ssm-hybrid.md) |47| Graph, GNN, message passing, node/edge features, or sparse graph workload appears. | [graph message passing runbook](references/runbook-graph-message-passing.md) |48| Classic CV detection, segmentation, keypoints, depth, OCR, or non-generative vision appears. | [non-generative CV runbook](references/runbook-non-generative-cv.md) |49| Time-series, forecasting, tabular sequence, anomaly detection, or temporal model appears. | [time-series forecasting runbook](references/runbook-time-series-forecasting.md) |5051Re-consult the map on a new config, parity failure, performance complaint, or52publish request.5354## Non-negotiable rules55561. **Do not execute untrusted model code during intake.** Inspect JSON, safetensors headers, source files, and licenses statically. Treat `auto_map`, custom modules, install hooks, and `trust_remote_code` as review gates.572. **Pin the source.** Record repository, revision, model files, tokenizer/processor revision, license, and checksum or artifact manifest.583. **Build a source oracle before porting.** Freeze deterministic fixtures and capture intermediate tensors at meaningful boundaries.594. **Port the smallest eager path first.** No quantization, compilation, custom kernels, batching, or speculative decoding until basic parity passes.605. **Change one optimization dimension at a time.** Keep a measurement and rollback record.616. **Prefer native MLX operations.** Try built-in fused operations, layout changes, cache design, and `mx.compile` before a custom Metal kernel.627. **Do not translate CUDA folklore mechanically.** A CUDA technique is only a research candidate until its Metal/MLX bottleneck and implementation are demonstrated.638. **Never hide quality regressions behind throughput.** For audio, language, vision, and generative models, use task-specific quality checks in addition to tensor tolerances.649. **Do not publish converted weights without license and provenance checks.** Preserve the original model card, attribution, generation config, tokenizer/processor files, and conversion recipe.6510. **Daily research automation is review-only.** It may collect and rank candidates, but must not silently rewrite runbooks or merge recommendations. Use the promotion-review ledger to separate review-ready findings from validation backlog and rejected leads.6611. **Experimental approaches require explicit opt-in.** Label unvalidated contributor, blog, paper, or repository learnings as experimental, state the missing validation gate, and ask before executing them: “This is an experimental approach. Do you want to try it?” Continue only if the user says to try it.6768## Workflow6970### 1. Inspect and classify7172For source models, run `scripts/inspect_model.py`, then `scripts/recommend_optimizations.py`, then `scripts/make_port_plan.py --artifact-root MODEL --recommendations ...`. An actionable plan re-inspects those bytes and recomputes the full recommendation report before advice; a blocked inspection yields only a remediation plan, and no override or registry read bypasses its blockers. An override may reorder a hybrid route but must preserve every routed family, runbook, and trait. For existing MLX projects or running ports, run `scripts/inspect_mlx_project.py` and read [inspector mode](references/inspector-mode.md) first.7374Read [intake and routing](references/intake-and-routing.md). Confirm source, risk, Mac, memory, performance, quality; record ambiguity in `PORT_PLAN.md`. For advice, read [model advisor playbook](references/model-advisor-playbook.md) and separate results into its five controlled advisor buckets. Numeric output may come only from `assets/effective_claims.json`; missing gates withhold the number.7576### 2. Select the closest proven MLX reference7778Consult [model support map](references/model-support-map.md) and `assets/architectures.yaml`. Prefer official MLX, Apple projects, pinned third-party MLX evidence, paper-only candidates, then new code; keep support scope explicit and verify config/layout.7980### 3. Establish the source oracle8182Run `scripts/capture_oracle.py` against the pinned local Hugging Face model before the MLX graph, then follow [parity and testing](references/parity-and-testing.md). It records inputs, embeddings, blocks, final norm, logits, hookable branches, and greedy IDs in a bounded NPZ plus manifest.8384### 4. Implement the minimal eager MLX graph8586Read [core porting method](references/porting-core.md); choose via the trigger map and the family records in `assets/architectures.yaml`. Load every runbook a hybrid route returns; the registry is the full inventory—never substitute an abbreviated list.8788Scaffold an unblocked dense decoder, sparse-MoE decoder, BERT encoder, T5 encoder-decoder, HuBERT/Wav2Vec2 acoustic encoder, or opt-in `minimal_selective` SSM:8990```bash91python3 scripts/scaffold_port.py inspection.json --artifact-root MODEL --output mlx_port92```9394It re-inspects and fails closed; review unsupported config, never patch around a generator blocker.9596Start eager: FP, batch one unless intrinsic, no compile/kernels, state/cache, assertions, reversible map.9798### 5. Convert weights deterministically99100Draft with `scripts/convert_checkpoint.py --emit-draft-map`, resolve the101schema-2 `WEIGHT_MAP`, validate with `scripts/validate_weight_map.py`, then102convert. Reject shard, coverage, shape, draft, or unresolved gaps; never mask103exceptions.104105### 6. Pass the parity ladder106107After conversion, `scripts/run_parity.py --mode MODE` runs source/MLX capture108and stops at the first failed rung. Modes: `dense-decoder` (default), `encoder`,109`encoder-decoder`, `ssm`, `asr`. `capture_mlx.py` retains captures,110`compare_tensors.py` extras, `_capture_common.py` bounded rules.111112When parity fails, use [failure atlas](references/failure-atlas.md); do not optimize a failing graph.113114### 7. Profile before choosing optimizations115116Read [benchmarking](references/benchmarking.md). Separate prefill, decode, postprocess, compile, memory, movement, sync, Python overhead; use `scripts/benchmark_command.py`.117118### 8. Apply the optimization ladder119120Use guides: [C](references/compile-and-kernels.md),[KV](references/attention-and-kv.md),[S](references/decoding-and-serving.md),[Q](references/quantization.md),[T](references/training-and-finetuning.md),[CS](references/compound-stacks.md).121122Tiers: `assets/recommendation-taxonomy.yaml`; for an experimental approach, state the gate and use the rule-11 opt-in prompt.123124Post-parity: one dimension; Metal only after proven bottleneck. Record hypothesis, gates, metrics, decision.125126### 9. Package and publish127128Follow [packaging and publication](references/packaging-and-publication.md). Include source, conversion, versions, artifacts, quantization, smoke, benchmarks, limits, license/attribution; no unsupported “faster” wording.129130### 10. Return an engineering report131132Summarize architecture/runbook, source/evidence, implementation/weights, parity, metrics, optimizations, risks, commands, and artifacts. Use `assets/`.133134## When to stop135136Stop when license, remote code, unresolved parity, regressions, missing kernel fallback/tests, or missing performance metadata blocks the result.137138## Maintenance139140[maintenance](references/maintenance-and-provenance.md), [hypothesis-led learning](references/hypothesis-led-learning.md), [research loop](references/deep-research-loop.md), [evidence graph](references/evidence-graph.md). Run `scripts/audit_skill.py --strict` and `scripts/validate_sources.py` before release.141Keep `tests/test_scenarios.py` at full family coverage when `assets/architectures.yaml` changes.
Amal-David