Skill Distiller
Transform skills authored for high-capability models (Opus) into deterministic workflows
that execute reliably on lower-cost models (Sonnet, Haiku). The core insight from
EvoSkills: skills encode reusable task structure, not model-specific artifacts. A skill
evolved on Opus transfers with +35-45pp gains to other models — but only when the
instructions are sufficiently deterministic that lower-capability models can follow them
without improvising.
Reference Files
| File |
Contents |
Load When |
references/distillation-patterns.md |
Pattern catalog for converting reasoning to rules |
Always |
Prerequisites
- The source skill must exist and satisfy the current
package-evaluator 2.0.0 contract: actual report-v2 static_conformance.status: PASS AND no unresolved CRITICAL or HIGH authored findings. Keep machine evidence and authored findings separate; live_effectiveness.status remains NOT_MEASURED.
- Access to both the source model (Opus) and target model (Haiku/Sonnet) for validation
- The
surrogate-verifier skill for cross-model assertion checking
- If the evaluator or required authored review is unavailable, report the missing prerequisite and stop before trace collection; do not fabricate a score or skip this gate. This prerequisite does not establish cross-model behavioral performance.
Workflow
Phase 1: Complexity Analysis
Score each section of the source SKILL.md for reasoning difficulty:
| Complexity Signal |
Score |
Distillation Action |
| Decision tree with 3+ branches |
HIGH |
Convert to explicit if/then lookup table |
| "Use judgment" or "consider context" |
HIGH |
Replace with concrete heuristic rules |
| Multi-step inference chain |
HIGH |
Break into numbered atomic steps |
| Reference to domain expertise |
MED |
Add explicit reference file with knowledge |
| Clear enumerated steps |
LOW |
Keep as-is |
| Concrete examples with expected output |
LOW |
Keep as-is |
Produce a complexity map: section name -> complexity score -> planned action.
Phase 2: Trace Collection
Execute the source skill with Opus on 5 representative tasks:
- Select tasks from
evals/cases.yaml (positive cases) or generate new ones
- For each task, capture the full execution trace:
- Tool calls made (which tools, in what order)
- Intermediate reasoning visible in output
- Final output structure and content
- Time taken and token usage
- Store traces as structured data for pattern extraction
Phase 3: Pattern Extraction
From the collected traces, extract deterministic patterns:
- Decision paths — For each HIGH-complexity section, find the actual decisions Opus made across
the 5 tasks. If Opus chose the same path in 4/5 cases, that path becomes the default rule
- Lookup tables — Where Opus applied domain knowledge, build explicit lookup tables
(e.g., "if input contains SQL, use these patterns; if input contains Python, use those")
- Concrete examples — Extract representative input/output pairs from traces to serve as
few-shot examples in the distilled skill
- Tool sequences — Identify the common tool invocation pattern and make it explicit
("Step 1: Read the file. Step 2: Grep for pattern X. Step 3: Write output.")
Phase 4: Distilled Rewrite
Rewrite the SKILL.md applying all distillation actions from Phase 1:
| Source Pattern |
Distilled Replacement |
| "Analyze the code and determine..." |
"Check for these 5 specific patterns: [list]" |
| "Use appropriate formatting" |
"Output as a markdown table with columns: [A, B, C]" |
| "Consider the context to decide..." |
"If [condition A]: do X. If [condition B]: do Y. Default: Z" |
| "Apply best practices for..." |
Reference file with explicit best practices enumerated |
| Multi-paragraph reasoning instruction |
Numbered step list with single-sentence steps |
Rules for the rewrite:
- Every instruction must be actionable by a model with no domain expertise
- No step should require inference — each step's input and output must be explicit
- Replace all "consider", "analyze", "determine" verbs with "check", "count", "list", "output"
- Add concrete examples for any step that could be ambiguous
- Keep the SKILL.md under 500 lines (distillation should reduce, not expand)
Phase 5: Target Model Validation
Run the distilled skill on the target model (Haiku or Sonnet):
- Execute the same 5 tasks from Phase 2 with the distilled skill loaded
- Use the
surrogate-verifier to generate assertions for each task output
- Compare pass rates:
| Metric |
Source (Opus + original) |
Target (Haiku + distilled) |
Delta |
| Assertions passed |
N/M |
N/M |
± |
| Weighted score |
X.XX |
X.XX |
± |
| Output completeness |
% |
% |
± |
| Format compliance |
% |
% |
± |
- If target model score < 80% of source model score, iterate:
- Identify which assertions the target model fails
- Add more explicit instructions for those specific failure points
- Re-run validation (max 3 iterations)
Phase 6: Cross-Model Report
Produce the final comparison:
# Skill Distillation Report: <skill-name>
## Complexity Reduction
- Sections distilled: N/M (HIGH → LOW)
- Instruction word count: original X → distilled Y (Z% reduction)
- Decision points replaced with lookup tables: N
## Cross-Model Performance
| Model | Assertions Passed | Weighted Score | Format Compliance |
|---------|-------------------|----------------|-------------------|
| Opus | 7/7 | 1.00 | 100% |
| Sonnet | 6/7 | 0.92 | 100% |
| Haiku | 5/7 | 0.85 | 85% |
## Changes Made
1. [Section] "Analyze complexity" → explicit 5-item checklist
2. [Section] "Apply formatting" → fixed markdown table template
...
## Recommendation
[SHIP | ITERATE | MANUAL_REVIEW_NEEDED]
Error Handling
| Error |
Resolution |
| Source static conformance FAIL or unresolved CRITICAL/HIGH authored findings |
Refuse distillation; report each evidence source separately and recommend source repair via test-engineer |
| Source evaluator or authored review unavailable |
Report the exact missing prerequisite; withhold gate completion and trace collection |
| No execution traces available |
Generate synthetic tasks and collect traces before proceeding |
| Target model fails all assertions |
Skill may be too complex for target model; report with detail |
| Distilled skill longer than source |
Review distillation; patterns may need consolidation |
Limitations
- Cannot distill skills that rely on open-ended adaptive reasoning at many decision points or multi-turn reasoning
- Visual/interactive skills (HTML generation, browser automation) may not distill well
- Distillation optimizes for determinism, not creativity — skills requiring open-ended generation
(writing, brainstorming) are poor candidates
- Trace collection requires actual model execution, incurring API costs
1---2name: skill-distiller3description: 'Converts Opus-quality skills into deterministic Haiku-executable workflows via trace-driven distillation and cross-model validation. Triggers on: "distill this skill", "make this skill work on Haiku", "cross-model optimization", "optimize skill for cost". NOT for code simplification, use code-refiner.'4metadata:5 source: https://skillmd.com/skills/mathews-tom/skill-distiller6 version: 1.0.27 category: development8 tags: [distillation, cross-model, optimization, haiku, deterministic]9 difficulty: advanced10 phase: build11---1213# Skill Distiller1415Transform skills authored for high-capability models (Opus) into deterministic workflows16that execute reliably on lower-cost models (Sonnet, Haiku). The core insight from17EvoSkills: skills encode reusable task structure, not model-specific artifacts. A skill18evolved on Opus transfers with +35-45pp gains to other models — but only when the19instructions are sufficiently deterministic that lower-capability models can follow them20without improvising.2122## Reference Files2324| File | Contents | Load When |25| -------------------------------------- | -------------------------------------------------- | ------------------------------------ |26| `references/distillation-patterns.md` | Pattern catalog for converting reasoning to rules | Always |2728## Prerequisites2930- The source skill must exist and satisfy the current `package-evaluator` 2.0.0 contract: actual report-v2 `static_conformance.status: PASS` AND no unresolved CRITICAL or HIGH authored findings. Keep machine evidence and authored findings separate; `live_effectiveness.status` remains NOT_MEASURED.31- Access to both the source model (Opus) and target model (Haiku/Sonnet) for validation32- The `surrogate-verifier` skill for cross-model assertion checking33- If the evaluator or required authored review is unavailable, report the missing prerequisite and stop before trace collection; do not fabricate a score or skip this gate. This prerequisite does not establish cross-model behavioral performance.3435## Workflow3637### Phase 1: Complexity Analysis3839Score each section of the source SKILL.md for reasoning difficulty:4041| Complexity Signal | Score | Distillation Action |42| ------------------------------------- | ----- | -------------------------------------------- |43| Decision tree with 3+ branches | HIGH | Convert to explicit if/then lookup table |44| "Use judgment" or "consider context" | HIGH | Replace with concrete heuristic rules |45| Multi-step inference chain | HIGH | Break into numbered atomic steps |46| Reference to domain expertise | MED | Add explicit reference file with knowledge |47| Clear enumerated steps | LOW | Keep as-is |48| Concrete examples with expected output| LOW | Keep as-is |4950Produce a complexity map: section name -> complexity score -> planned action.5152### Phase 2: Trace Collection5354Execute the source skill with Opus on 5 representative tasks:55561. Select tasks from `evals/cases.yaml` (positive cases) or generate new ones572. For each task, capture the full execution trace:58 - Tool calls made (which tools, in what order)59 - Intermediate reasoning visible in output60 - Final output structure and content61 - Time taken and token usage623. Store traces as structured data for pattern extraction6364### Phase 3: Pattern Extraction6566From the collected traces, extract deterministic patterns:67681. **Decision paths** — For each HIGH-complexity section, find the actual decisions Opus made across69 the 5 tasks. If Opus chose the same path in 4/5 cases, that path becomes the default rule702. **Lookup tables** — Where Opus applied domain knowledge, build explicit lookup tables71 (e.g., "if input contains SQL, use these patterns; if input contains Python, use those")723. **Concrete examples** — Extract representative input/output pairs from traces to serve as73 few-shot examples in the distilled skill744. **Tool sequences** — Identify the common tool invocation pattern and make it explicit75 ("Step 1: Read the file. Step 2: Grep for pattern X. Step 3: Write output.")7677### Phase 4: Distilled Rewrite7879Rewrite the SKILL.md applying all distillation actions from Phase 1:8081| Source Pattern | Distilled Replacement |82| -------------------------------------- | ------------------------------------------------------------ |83| "Analyze the code and determine..." | "Check for these 5 specific patterns: [list]" |84| "Use appropriate formatting" | "Output as a markdown table with columns: [A, B, C]" |85| "Consider the context to decide..." | "If [condition A]: do X. If [condition B]: do Y. Default: Z" |86| "Apply best practices for..." | Reference file with explicit best practices enumerated |87| Multi-paragraph reasoning instruction | Numbered step list with single-sentence steps |8889Rules for the rewrite:90- Every instruction must be actionable by a model with no domain expertise91- No step should require inference — each step's input and output must be explicit92- Replace all "consider", "analyze", "determine" verbs with "check", "count", "list", "output"93- Add concrete examples for any step that could be ambiguous94- Keep the SKILL.md under 500 lines (distillation should reduce, not expand)9596### Phase 5: Target Model Validation9798Run the distilled skill on the target model (Haiku or Sonnet):991001. Execute the same 5 tasks from Phase 2 with the distilled skill loaded1012. Use the `surrogate-verifier` to generate assertions for each task output1023. Compare pass rates:103104| Metric | Source (Opus + original) | Target (Haiku + distilled) | Delta |105| ------------------------------- | ------------------------ | -------------------------- | ----- |106| Assertions passed | N/M | N/M | ± |107| Weighted score | X.XX | X.XX | ± |108| Output completeness | % | % | ± |109| Format compliance | % | % | ± |1101114. If target model score < 80% of source model score, iterate:112 - Identify which assertions the target model fails113 - Add more explicit instructions for those specific failure points114 - Re-run validation (max 3 iterations)115116### Phase 6: Cross-Model Report117118Produce the final comparison:119120```markdown121# Skill Distillation Report: <skill-name>122123## Complexity Reduction124- Sections distilled: N/M (HIGH → LOW)125- Instruction word count: original X → distilled Y (Z% reduction)126- Decision points replaced with lookup tables: N127128## Cross-Model Performance129| Model | Assertions Passed | Weighted Score | Format Compliance |130|---------|-------------------|----------------|-------------------|131| Opus | 7/7 | 1.00 | 100% |132| Sonnet | 6/7 | 0.92 | 100% |133| Haiku | 5/7 | 0.85 | 85% |134135## Changes Made1361. [Section] "Analyze complexity" → explicit 5-item checklist1372. [Section] "Apply formatting" → fixed markdown table template138...139140## Recommendation141[SHIP | ITERATE | MANUAL_REVIEW_NEEDED]142```143144## Error Handling145146| Error | Resolution |147| ----------------------------------- | ------------------------------------------------------------- |148| Source static conformance FAIL or unresolved CRITICAL/HIGH authored findings | Refuse distillation; report each evidence source separately and recommend source repair via test-engineer |149| Source evaluator or authored review unavailable | Report the exact missing prerequisite; withhold gate completion and trace collection |150| No execution traces available | Generate synthetic tasks and collect traces before proceeding |151| Target model fails all assertions | Skill may be too complex for target model; report with detail |152| Distilled skill longer than source | Review distillation; patterns may need consolidation |153154## Limitations155156- Cannot distill skills that rely on open-ended adaptive reasoning at many decision points or multi-turn reasoning157- Visual/interactive skills (HTML generation, browser automation) may not distill well158- Distillation optimizes for determinism, not creativity — skills requiring open-ended generation159 (writing, brainstorming) are poor candidates160- Trace collection requires actual model execution, incurring API costs