Trace-to-Skill
Evolve agent/skill definitions by analyzing real execution traces across diverse tasks. Parametric knowledge (asking LLM to write skills from imagination) = useless. Must ground in real traces.
Why This Works
Parallel fleet of analysts examine diverse traces → extract patches → hierarchical merge keeps only patterns recurring across many traces → those are systematic, not noise. One comprehensive skill > fragmented collection.
Pipeline
SPECS → TRACES → ANALYSTS → MERGE → APPLY → VALIDATE
15 15 4×batch 1 patch held-out
diverse parallel parallel merge agents 5-7 specs
specs sessions analysts report + docs verify
Step 1: Spec Pool (15 diverse features)
Cover every dimension the target agent handles. Narrow pool = narrow skill.
| Dimension | Spread Required |
|---|---|
| Size | 1 XS, 4-5 S, 7 M, 2 L |
| Patterns | Every pattern the agent should know |
| Complexity | Simple CRUD → multi-step workflows → external APIs |
| Auth/RLS | User-scoped, admin-only, multi-tenant |
| Edge cases | Real-time, file upload, Stripe, Inngest |
Write coverage matrix in README.md. Every dimension hit by 2+ features.
Step 2: Workspace Setup
dev/{phase}-stress-test/
├── README.md # Coverage matrix + status
├── SESSION-INSTRUCTIONS.md # Frozen rules for trace sessions
├── roadmap.json # Claim/lock/status tracking
├── specs/{NN}-{name}.md # 15 pre-written specs
├── traces/{NN}-{name}/ # Full execution outputs
│ ├── {phase-outputs} # plans, migrations, code, etc.
│ └── decisions.md # Struggles = GOLD
├── analysts/
│ ├── {analyst-type}.md # 4 analyst definitions
│ └── merge-prompt.md # Merge coordinator
└── analysis/
├── {analyst}-patches-b{N}.md # Raw patches per batch
└── evolution-report.md # Final merged output
Step 3: Generate Traces (1 per session)
Each session: claim feature → read spec → run EXACT production process → save outputs.
Rules:
- ONE feature per session (no cross-contamination)
- EXACT production process (no shortcuts)
- Freeze agent definition S0 (same version for ALL traces)
- Document struggles in decisions.md (richest signal)
- Claim-lock via git commit+push if parallel sessions
Run 5-8 sessions in parallel. 15 features ÷ 5 parallel ≈ 3 rounds.
Step 4: Analyze (4 Analysts × Batches of 4-5)
Four specialized analysts run in parallel per batch. Each sees batch traces + frozen S0. Analysts NEVER see each other's patches.
| Analyst | Focus | Patch Prefix |
|---|---|---|
| Architecture | Cache, RLS, boundaries, data flow, Inngest | ARCH_PATCH |
| Pattern | Pattern selection, compliance, anti-patterns, brief quality | PATTERN_PATCH |
| Completeness | Missing UI states, mutation flows, mobile, a11y, edge cases | COMPLETENESS_PATCH |
| Buildability | Sequencing, handoffs, task clarity, brief sufficiency, risks | BUILD_PATCH |
Patch format:
{PREFIX}: {feature-id}
AREA: {specific area}
GAP: {what's missing/wrong}
PATTERN: {when this happens — GENERALIZE}
FIX: {concrete fix}
SEVERITY: {critical | important | minor}
Batch 4-5 features per round (not all 15 — analyst needs cross-trace comparison within batch).
Adapt analysts per phase — see references/phase-analysts.md.
Step 5: Merge (Prevalence-Weighted)
Consolidate ALL patches across all analysts and batches into one evolution report.
| Frequency | Routing |
|---|---|
| 8+ features | Universal → agent definition checklist. MUST add. |
| 4-7 features | Common → agent def or pattern doc. SHOULD add. |
| 2-3 features | Occasional → brief template or reference. Consider. |
| 1 feature | Idiosyncratic → discard (unless severity=critical). |
Merge guidelines:
- Deduplicate — same gap from different analysts → keep best version
- Resolve conflicts — contradictory → pick stronger evidence or synthesize
- Preserve unique insights — different failure modes → keep all
- Prevalent pattern bias — recurring = systematic, express as general principles
- Be concrete — "always verify X after Y" > "be thorough about Z"
- Max 30 new rules per round
Output categories:
| Output | Where Applied |
|---|---|
| Agent checklist items | .foundry/agents/{agent}.md |
| Brief template additions | Brief templates in agent defs |
| Pattern doc gaps | .foundry/references/source-docs/patterns/ |
| Sizing corrections | Orchestrator mode rules |
Step 6: Apply + Validate
- Write evolution-report.md with diff-ready patches
- Apply patches to agent definitions + pattern docs
- Run held-out specs (5-7 not used in analysis) with evolved agents
- Compare pre vs post quality
- If gaps remain → iterate (2-3 rounds converge)
Phase Roadmap
Each phase reuses the same 15 feature specs, evolving different agents:
| Phase | Target Agents | Trace Artifacts | Key Analyst Adaptations |
|---|---|---|---|
| Planning | planner | plans, briefs, decisions | Standard 4 analysts |
| DB | db-architect | migrations, RLS, indexes | Schema analyst, RLS analyst |
| Backend | backend + backend-evaluator | procedures, Inngest, evaluator grades | Export shape, error handling, evaluator calibration |
| Frontend | connector + designer + design-evaluator | pages, components, evaluator grades | RSC boundaries, all UI states, evaluator calibration |
| Testing | api-tester, flow-tester | tests, results, bug fixes | Coverage, reliability, fixtures |
| Full E2E | orchestrator + all agents | entire pipeline output | + Pipeline analyst (handoff gaps) |
Order: Evolve each phase individually first, then run full E2E stress test with all evolved agents. Each phase feeds evolved outputs to the next (evolved plans → DB traces, evolved migrations → backend traces).
Key Principles
- Parametric = useless. Must have real traces with real failures.
- Error analysis > success analysis. Errors reliably valuable, successes volatile.
- Parallel > sequential. All analysts see frozen S0, no drift, 5-20× faster.
- Prevalence = generalizability. Recurring across diverse traces = systematic.
- Struggles are gold. decisions.md doubts/tensions = richest trace artifact.
- Max 30 rules/round. Too many = none followed.
- Skills transfer downward. Opus-evolved skills improve Sonnet, not always vice versa.
- Diversity > volume. 15 diverse specs > 50 similar ones.
- Separate generator from evaluator. Agents can't judge their own work. (Anthropic research)
- Evaluator calibration > generator checklists. Easier to tune a skeptical judge than a self-critical builder.
- Strip scaffolding per model. Re-examine what's load-bearing on each model release.
- Planners stay high-level. Define WHAT (contracts), not HOW (implementation). Over-specification cascades.
Evaluator Calibration (NEW — for Backend + Frontend phases)
When stress-testing phases that include evaluators (backend, frontend), add a calibration loop:
- Run traces with evaluator in the pipeline
- Read evaluator grades — find where evaluator judgment diverges from yours:
- Evaluator said PASS but you'd say FAIL (too lenient)
- Evaluator said FAIL but you'd say PASS (too strict)
- Evaluator missed a violation entirely (blind spot)
- Update evaluator prompt with calibration examples (few-shot)
- Re-run 3 traces with calibrated evaluator
- Repeat until evaluator grades match your judgment on 80%+ of criteria
This is the paper's key insight: tuning a standalone evaluator to be skeptical is more tractable than making a generator critical of its own work. The calibration loop IS the stress test for evaluators.
Evaluator trace artifacts (in addition to standard):
- Grade report (the 4-criteria table)
- Fixes applied (diff of changes)
- False positives (evaluator flagged something that was actually correct)
- False negatives (evaluator missed something you'd catch)
Anti-Patterns
| Don't | Why | Do |
|---|---|---|
| Analyze all 15 at once | Too much per analyst | Batch 4-5 |
| Generic analyst ("find problems") | Shallow patches | Specialized + checklisted |
| Skip decisions.md | Miss richest signal | Require struggles documented |
| Merge without frequency | Can't filter noise | Count citations per pattern |
| Patch agents directly | Lose rationale | Evolution report first |
| Reuse trace specs for validation | Overfitting | Hold out 5-7 specs |
| Evolve all phases at once | Can't attribute improvement | One phase → validate → next |
References
references/phase-analysts.md— Analyst adaptations per pipeline phasereferences/merge-prompt.md— Merge coordinator prompt templatereferences/session-template.md— Session instructions template