fudesign2008 Distill Session — Extract What You Can't Yet Explain
The most valuable knowledge in a top-tier model's output is the knowledge it applied without mentioning. This skill makes that invisible reasoning visible, testable, and transferable.
The Core Insight
When a weaker model fails, the gap is almost never "it didn't try hard enough." The gap is that the top-tier model executed steps it doesn't consciously report — constraint checks it ran silently, orderings it chose without deliberation, failure modes it avoided by reflex. These implicit behaviors are the extraction target.
The hard part is not getting the top-tier model to succeed. It's getting it to notice what it did that was non-obvious.
When to Use
- A weaker model produced wrong or shallow output on a task you handle well
- You completed a complex task and want to mine it for transferable rules
- A skill (coding-workflow, debug-workflow, task-router) needs strengthening
- You notice yourself doing something you can't fully articulate yet
Distill immediately after completing a complex task — don't wait for a failure to trigger this. Your reasoning is still warm.
Phase 1 — CAPTURE
Goal: Preserve exact failure and success before analysis distorts them.
Save to .omc/distill-sessions/YYYY-MM-DD-[short-title].md:
# Distill: [short title]
Date: YYYY-MM-DD | Task type: code-change / bug-fix / refactor / review
## The Task
[Exact prompt — verbatim]
## Weak Model Output
[Exact output — do NOT paraphrase. Preserve reasoning chain if visible.]
## What Went Wrong (Observable)
[Concrete defects: wrong file, missing call site, incorrect assumption.]
## Top-Tier Model Output
[Your actual output.]
Critical: Paste the weak model's FULL output. Phase 3 requires comparing actual text, not your memory of it.
Phase 2 — THE EXTRACTION (Core of This Skill)
Goal: Surface the implicit reasoning that made the difference.
This is where most distillation fails. Generic "what steps did you take?" gets cleaned-up narratives, not actual reasoning. The questions below bypass post-hoc rationalization to reach real decision points.
The Extraction Prompt
Answer with ruthless honesty. Do not clean up your reasoning or make it sound more systematic than it was.
<extraction>
## 1. What did I look at before I started doing anything?
[Specific files, symbols, docs you read BEFORE any change. Include
things you glanced at and decided weren't relevant. Weaker models
start acting before this step is complete.]
## 2. What did I almost do wrong, then correct?
[Any moment you began one approach, noticed a problem, pivoted.
These near-misses reveal constraints you check implicitly. A weaker
model would have followed through on the wrong approach.]
## 3. What information was I holding in my head simultaneously?
[Facts, constraints, relationships tracked at the same time. Example:
"function X called from 3 places, return type used as discriminated
union downstream, one call site in a hot path." Weaker models handle
these serially or drop some.]
## 4. Where did I choose to NOT do something?
[Restraint is invisible. Skipped refactors? Edge cases not worth
handling? Suppressed "while I'm here" changes? Weaker models often
fail by doing too much, not too little.]
## 5. What ordering did I choose and why?
[Dependency reason? Safety reason? Would a different order cause
problems? Weaker models edit in prompt-order rather than
dependency-order.]
## 6. What would break if I removed any single step?
[Steps where removal causes failure are essential. Steps where
removal doesn't matter were optional.]
## 7. What is the simplest incorrect version of my output?
[The most plausible wrong answer — ~80% understanding of the task.
Usually what the weak model produced. Name the specific 20% gap.]
## 8. Write the instruction set (3-7 rules)
[Each rule must be:
- Actionable in a single step (no "think carefully about X")
- Verifiable by a third party (someone can check if it was done)
- Triggered by an observable condition (not "when appropriate")]
</extraction>
Why These Questions Work
| Question | Surfaces | Generic prompts miss |
|---|---|---|
| What I looked at first | Exploration scope | Models report actions, not reads |
| What I almost did wrong | Live constraint checking | Post-hoc narratives erase near-misses |
| Info held simultaneously | Working memory load | Serial descriptions hide parallel reasoning |
| Chose NOT to act | Restraint, scope control | Success stories omit inaction |
| Ordering chosen | Dependency awareness | "I did A then B" without why |
| What breaks without step | Essential vs. optional | Models pad with extra steps |
| Simplest wrong version | The actual failure mode | "What could go wrong" too open-ended |
Phase 3 — DELTA ANALYSIS (Most Error-Prone Phase)
Goal: Identify which behaviors CAUSED the quality difference.
The common mistake: attributing success to the most impressive-looking step rather than the step that actually mattered.
The Attribution Test
For each behavior from Phase 2:
If the weak model did THIS ONE THING, would it have succeeded? Yes → candidate root cause. No → contributing factor at best.
Did the weak model lack ABILITY or PROMPTING? Ability gap → needs a tool, not a rule. Prompt gap → extractable.
Specific to this task, or generalizable? Specific → useful as example. General → candidate pattern.
Common Mistakes
| Mistake | Fix |
|---|---|
| "I understood better" | What SPECIFIC thing did you read that the weak model didn't? |
| Sequence ≠ causation | Would swapping the order alone have fixed it? |
| Overgeneralizing ("read everything") | WHICH files? What was the selection criterion? |
| Undergeneralizing ("check discriminated unions") | General principle: "check downstream consumers of modified types" |
| Survivorship bias | Remove each step mentally. Which removals cause failure? |
Output
## Delta Analysis
Root cause: the weak model failed because it ___
Minimum fix: [single most impactful rule]
Contributing factors: 1. [behavior] — [impact] 2. [behavior] — [impact]
Non-essential steps I took: [prevents rule bloat]
Phase 4 — DISTILL INTO RULES
Every rule must pass ALL FIVE criteria:
| Criterion | Bad | Good |
|---|---|---|
| Actionable (one step) | "Be thorough" | "Grep for all call sites before modifying" |
| Verifiable (checkable) | "Consider edge cases" | "List 3 edge cases, trace each through code" |
| Triggered (when exactly) | "When modifying code" | "When adding a parameter to an exported function" |
| Minimal (no extra words) | Paragraph with examples | One sentence, one action |
| Causal (addresses root cause) | "Read more code" | "Read all files that import the module you're changing" |
The causality check: "If the weak model followed ONLY this rule and changed nothing else, would the output improve?" Yes → keep. No → delete. Maybe → make more specific.
Save to .omc/distill-sessions/patterns/[task-type].md:
### Rule [N]: [short name]
**Trigger**: [Observable condition]
**Action**: [Single concrete instruction]
**Because**: [What goes wrong without it]
**Test**: [How to verify it was followed]
Phase 5 — INTEGRATE
Propagate rules into the skills that govern the behavior:
| Rule targets... | Update... |
|---|---|
| What to read before coding | coding-workflow Phase 1 |
| Change ordering | coding-workflow Phase 2 |
| Implementation discipline | coding-workflow Phase 3 |
| Verification | coding-workflow Phase 4 |
| Root cause identification | debug-workflow / systematic-debugging |
| Task classification | task-router |
Bump target skill version (patch: 1.0.0 → 1.0.1).
Phase 6 — VALIDATE
The real question: Does this rule produce CORRECT output, or just BETTER-LOOKING output?
"Explain your reasoning before acting" makes outputs look thoughtful without improving correctness. "Grep for all call sites before modifying a function signature" produces measurably different output. Target the latter.
- Re-run EXACT original task with weaker model + updated skill
- Compare on the specific dimension that failed, not overall quality
## Validation
Original failure: [what was wrong]
After distillation: [is that specific thing correct now?]
VERDICT: FIXED / BETTER BUT STILL WRONG / NO CHANGE / REGRESSION
If not FIXED — was Phase 3 wrong? Re-examine root cause.
BETTER BUT STILL WRONG → rule addresses symptom, not cause. Redo Phase 3. NO CHANGE → rule not triggered, or action beyond model capability.
Quick Reference: Copy-Paste Extraction Prompt
The single most important artifact. Use immediately after completing a task.
You just completed a task. Before moving on, fill in this extraction
honestly — especially the parts that feel embarrassing or obvious.
<extraction>
## What I looked at before acting
## What I almost did wrong, then corrected
## What I was holding in my head simultaneously
## What I chose NOT to do
## What ordering I chose and why it mattered
## What would break if I skipped any single step
## The simplest plausible wrong answer to this task
## Rules for a weaker model (3-7, each one sentence, each verifiable)
</extraction>