Analyze

Analyze — reading trajectories, extracting experience

Your input is a directory of <task_id>.md files, one per task in this round. No pre-computed summaries, no cluster files — you investigate the markdown directly. Your output is a set of candidate hypotheses, each naming a concrete pattern (either a gap to close or a habit to preserve / generalize) and the lever you'd pull.

Passing and failing trajectories are treated as equal evidence sources. A passing cluster can teach you a habit worth locking in or worth lifting into the template for the failing cluster; a failing cluster can teach you a blocker worth closing. Both feed the same candidate pipeline — same template, same retroactive check, same global-first ordering. The bias to read only failures is the single biggest cause of "right diagnosis, still-net- negative round": fixes land but previously-working habits silently regress because no candidate preserved them.

Trajectory file shape

YAML frontmatter + markdown body. Read <path> limit=30 returns the frontmatter plus the first few body lines — usually enough to classify the task. When the frontmatter hints at a deeper cause, read the last ~40 lines of the body to see what the agent actually had in context at its decisive step.

Frontmatter tiers (render order)

Behaviour — always present

• exit_reason — done / max_steps / budget_exceeded / loop_detected / error
• steps, total_tokens, cost_usd, elapsed_s
• tool_call_counts — {tool_name: n}
• tool_error_counts — {tool_name: n}
• error_count, pivotal_tool
• final_output_length

Eval — always present, authoritative

• eval_passed (bool) — the external pipeline evaluator's verdict
• eval_score (float) — numeric score (0.0 / 1.0 for exact-match)

eval_* is the ground-truth correctness signal. When Eval and Judge disagree, believe Eval. The dataset's expected answer text is intentionally withheld from the frontmatter, and so is the evaluator's textual reason — you see only pass/fail and score, never the target string. Evolve on correctness signals, not on a known target. For the why behind a failure, read the trajectory body and the optional judge_* fields below.

Judge — optional

Present only when LLMJudgeProcessor ran. An LLM's opinion, not ground truth. Useful for why a task failed, not whether.

• judge_verdict — plausible / unsupported / hedging / format_wrong / refused / no_answer / judge_error
• judge_cause, judge_missing, judge_lesson — one-line strings
• judge_missing_capability — {present: bool, summary: "..."}; primary signal for Action-lever decisions
• extracted_answer — what the agent actually committed (the expected answer text is never exposed; do not try to reconstruct it)

judge_error is the judge failing (timeout, bad JSON), not the agent. Treat as "no opinion" and lean on behaviour + body.

The two axes of reflection

Every candidate hypothesis is pinned on two axes. Keep them independent — don't collapse the lens into the lever.

Axis A — Lens (what is the pattern?)

Three reading passes over the same batch, not three buckets. A single trajectory can answer more than one question — you are sweeping for signals, not assigning labels. Every pass anchors on the authoritative eval_passed field; behaviour / judge signals describe the shape, never the verdict.

exit_reason=done alone is not evidence of success — the agent merely called end_turn. Always gate on eval_passed before reading a trajectory as a positive example.

The three lenses feed the same candidate pipeline. A Success- lens observation turns into a candidate in one of two shapes:

• Lock-in — encode the habit as Configuration / Control / Instruction so a future round's edit cannot silently regress it. Justification: the cited passing tasks would fail if the habit were removed.
• Transfer — lift the habit into a rule applied to a failing cluster. Justification: the cited failing tasks would pass if the habit were applied.

Failure and Capability-gap lenses produce corrective candidates (close a gap). Both polarities go through the same retroactive check, same global-first ordering, same candidates.md schema. What changes is which variant of the retroactive check the candidate has to answer.

Rushing to Failure and skipping the other two leaves Capability-gap bets (blocked-but-sometimes-recoverable shapes) and Success-pattern lessons (what to preserve or generalise) on the table.

Axis B — Lever (where does the fix belong?)

The axes are orthogonal. A Failure symptom can be fixed at any of the four levers. A Capability-gap lens does not always mean a new tool — if the existing tool fires but returns unusable data, the lever is often Control (an on_after_tool processor that parses), not Action.

Worked examples — lens × lever × intent

The three axes cover 3 × 4 × 3 combinations; most are rare. Below are representative shapes you'll actually see. Use them to pressure-test your own candidate — if yours doesn't resemble any row here, either you've found something genuinely new or the lens / lever / intent tag is wrong.

Rows omitted on purpose: Capability-gap / Instruction is almost always wrong (if the capability isn't there, no prompt rule conjures it — covered in "Action vs Instruction" below); Success / Configuration is rare because knob values don't encode a habit; Success / Action is rare because authoring a new tool is not a natural way to preserve a habit the existing tools already enable.

If your candidate lands in an omitted cell, that's a signal to challenge the lens / lever / intent tag before the retroactive check.

Read strategy

Your output is labelled clusters with cited evidence, not a fixed procedure run. The trajectories live under _meta_scratch/traj/ as <task_id>.md files. Available tools: Read / Grep for direct inspection, spawn_reflect_worker(kind="trajectory- digester", files=[...]) for batching when the task count would blow the token budget from TASK.md. Pacing — how many frontmatters to scan before clustering, how many representatives to body-read per cluster, when delegation is worth it — is your judgement call. Record the reasoning in candidates.md.

Two constraints are non-negotiable:

• Systemic vs idiosyncratic. A cluster (failing OR passing) earns a config change only when the same shape recurs across at least two distinct tasks with different inputs. One-off observations — however obvious the fix or however clean the habit feels — go to NEEDS_FROM_HUMAN and are skipped. Two tasks that look different but share the same root cause still count as one "gap" (test: if you fix one, does the other follow?); two tasks with coincidentally similar frontmatter but different mechanisms do not count as a cluster.
• Frontmatter shows the ceiling shape; the body shows what actually happened at the decisive step. Don't propose Action or Control on frontmatter signals alone — for failure candidates a body must name the actual failure mode (raw bytes, empty results, redirect HTML, cycling URL, silent parse failure); for success candidates a body must show the habit actually firing (the ## Plan block, the retry loop, the tool sequence) rather than just inferring it from short step count. A candidate whose evidence cites only a worker digest, with no trajectory body quoted, is not grounded.

The retroactive check — three variants, one per candidate intent

Before writing any code, for each candidate, ask the variant that matches the candidate's intent. A candidate has exactly one intent; one that seems to need more than one variant is two candidates glued together — split them.

Variant A — Corrective (Failure / Capability-gap lens)

If this fix had already been in place on the cited failing trajectories, would the task have succeeded?

• A commit-nudge requires an answer to commit.
• A template rule requires data to apply.
• A tighter loop detector requires the loop to be the actual blocker, not the consequence of the real blocker.
• A new tool requires the task to have needed that tool's output shape, not just the tool's topic.

When the answer is "no — the agent would still have failed", the symptom is downstream of an earlier failure. Trace one step back: what did the agent lack, not what did it fail to do? That upstream gap is the real lever. Drop or re-scope.

Variant B — Preservative, lock-in (Success lens)

If this pattern were removed from the current config, would the cited passing trajectories have failed?

• A habit that's already reliable "by accident" of the current prompt / tool / model combo is not worth encoding — it costs complexity and pays nothing. Drop.
• A habit that was visibly load-bearing (the body shows the failing task in an adjacent cluster not doing this habit and breaking) earns a lock-in candidate — typically Instruction (promote the habit to an explicit rule) or Control (a guard that enforces it). Regression-probe candidates live here.

Variant C — Preservative, transfer (Success → failing cluster)

If this pattern had been applied to the cited failing trajectories, would they have passed?

• The generalization must pass both ends: the passing cluster's body must show the habit firing, AND the failing cluster's body must show the habit absent at a moment where applying it would have unblocked the decisive step.
• If only the first end is grounded, the habit is domain-specific — you're about to force a passing-cluster quirk onto tasks it won't help. Drop or re-scope.

Why the variants matter

The single-question form ("would the fix have worked") silently excludes Success-lens candidates — they can't "work" on the tasks they came from, because those tasks already passed. Without Variant B/C, Success-lens evidence has no path into candidates.md, and the round ships fixes while quietly regressing what was already working.

Disambiguating adjacent levers

The Axis B table tells you what each lever is; the sections below help when two lever choices could explain the same evidence.

The first question — tool layer or caller?

Before disambiguating adjacent levers, ask where the blame sits: on the tool layer's return or on the caller's use of the return. This single split collapses most lever choices.

Points at the tool layer (Action or Control territory — weigh together, no single signal decisive):

• tool_error_counts[T] recurs across distinct tasks on tool T.
• judge_missing / judge_missing_capability keeps naming the same shape (missing client, absent parser, no fallback).
• An existing tool fires successfully but returns data the caller has no reasonable way to use (raw bytes, truncated page, aggregated when itemized is needed, unparseable response).

Points at the caller (Instruction or Configuration territory):

• Tool returned correct data; agent misread, didn't reach for it, or committed before using it.
• Failure reads as budget / commitment / loop behaviour rather than data availability.

Once you know which side is to blame, the two sections below disambiguate within that side. Configuration vs Instruction rarely cross-contaminate and don't need a section; they split naturally (existing-knob tune vs prompt-level rule).

Perceived difficulty gradient isn't real. Template < processor < tool feels true but once you've written one of each, all three are "write one file, wire it into config, verify". Pick by evidence, not by ease.

Action vs Instruction — when the gap is capability, not knowledge

Signals that a missing capability (not a missing prompt rule) is the gap:

• judge_missing_capability.summary keeps naming the same shape across distinct tasks.
• Recurring cross-task tool_error_counts on the same tool.
• An existing tool fires but returns data the caller has no reasonable way to use (raw bytes, truncated page, unparseable response).
• Your own failure-bucket phrasing becomes "can't access X", "can't parse Y", "doesn't reach Z" for a recurring input shape.

In this regime, reaching for another template rule is usually wrong — instruction patches around a tool that returns nothing useful tend to compound into more failure. Prefer Action. For the authoring mechanics (TOOL_SPEC.md, @tool signature, scoping a tool to a class of tasks) see the reference skill — come back here for lever choice, go there for how to build it.

Counter-case — when Instruction is still right:

• Evidence shows the capability is actually present (the answer was in a tool result the agent ignored, or judge is confused).
• Failure reads as commitment / format / timing, not data availability (e.g. judge_verdict=format_wrong on tasks whose reasoning was otherwise correct).

Document why not Action in the journal so the next round can reuse the reasoning.

Control vs Action — when post-processing beats a new tool

Tool fires and returns correct data, but in a shape the caller can't use directly (binary bytes instead of parsed text, raw HTML after a redirect chain, unstructured JSON blob). An on_after_tool processor that normalises the return is usually stronger than a new tool that does the same fetch-and-parse — duplicating the fetch loses the existing tool's robustness and budget accounting.

Control also covers input sanitisation (on_before_tool) and cross-task guards (on_step_start / on_before_model) that must fire uniformly across every task — shapes a per-call tool can't express at all.

Counter-case — when Control isn't enough: if the tool's fetch itself is wrong (wrong endpoint, missing auth, truncates before returning), no post-processor recovers what was never retrieved. Quick test: can the information the agent needs be reconstructed from what the tool currently returns? Yes → Control. No → Action.

When a prior round failed the replay gate

If the last round was rejected by the replay gate (_meta_scratch/REPLAY.md exists with ok: false), treat it as evidence not punishment. The JSON lists which tasks tripped (exit reason, first failing step, brief traceback). Before re-proposing anything:

• If the failure was structural (exit_reason=error, tool ImportError, Jinja render error) — the previous round's code was wrong, not its reasoning. The hypothesis may still be sound; re-scope the candidate and fix the implementation.
• If the failure was semantic (candidate ran cleanly but didn't flip the target task) — the retroactive check returned yes on paper but the real run disagreed. Treat the retroactive check as having failed retrospectively; don't re-ship the same shape without new body evidence.
• If the failure was a regression on a previously-passing task you didn't predict — that's a preservative-lock signal the prior round missed. Consider a Variant-B candidate next round to lock in whatever got disrupted.

The replay gate report is the highest-quality signal in the journal — it's from an actual run, not a judge opinion.

Cross-round signals

The journal CONTEXT.md renders a lever scoreboard. Use it as a prior on what's worth trying, not as a rule:

• A lever that has landed consistently is worth revisiting.
• A lever that has been tried repeatedly on the same cluster without flipping any task is a signal to look elsewhere, not a reason to try again harder.
• A reverted hypothesis_id cannot be re-proposed in the same shape without new evidence; different shape at a different lever is fair game.

Candidate ordering: global first, local second

Before choosing what to ship, rank candidates by global net value, not by the most dramatic local anecdote.

Apply this ordering:

1. Global gain first: prefer candidates that plausibly flip a recurring failing cluster (>=2 tasks) or that prevent a likely regression on a currently-passing cluster of similar size. A preservative-lock candidate with a plausible 2-task regression saved beats a corrective candidate with a 1-task speculative flip.
2. Regression surface second: demote candidates with meaningful risk to already-passing or unrelated clusters unless evidence is unusually strong. (Preservative-lock candidates are specifically reducing this risk — they should be weighted accordingly, not dismissed as "no flips".)
3. Cost shift third: among similar gain/risk profiles, prefer lower expected token/cost inflation.

Every shipped candidate should make the tradeoff explicit:

• expected_global_gain
• regression_risk
• cost_shift

If a candidate helps one target task but likely harms global pass-rate, drop it or re-scope it.

Multi-lever rounds

A round can ship several independent changes when the evidence warrants — one per distinct pattern (gap or habit). "Independent" means each change works on its own; no same-round dependency chains. Ship the number the evidence supports, not a target. Breadth without focus regresses pass_rate; depth without breadth misses cheap wins. A good round typically pairs 1-2 corrective candidates (closing gaps) with 0-1 preservative candidates (locking in or generalizing habits) — the exact mix is evidence- driven, not a quota.

_meta_scratch/candidates.md — required when the config changes

Before writing the config, draft your candidates in _meta_scratch/candidates.md. Each candidate is a section with a stable ID (C-001, C-002, …) that the journal entry's cited_candidates frontmatter will reference.

Header format — the orchestrator parses ^## Candidate C-\d+ headers, so every section header must match this shape. Anything else (numbered bullets, "Candidate 1", etc.) is invisible to the evidence gate and the round is rejected.

Required fields per candidate — each candidate must include all of the fields below. A candidate missing any of them is treated as under-grounded; the lever choice will be challenged and the round may be rejected at review even if the header format parses.

# Candidates

## Candidate C-001
[lens: capability-gap | lever: control | intent: corrective]

One-line fix description.

- Tasks affected: task-011, task-042, task-108
- Signal: `judge_missing` names output-parsing capability on all three
  tasks; `tool_error_counts[fetch_tool]=0` (fetch is fine — shape is
  not).
- Verified (Read): task-011 step 7 — tool returned raw unstructured
  output, agent described content vaguely. task-042 step
  5 — same shape. task-108 step 9 — same shape.
- Why Control not Action: the tool already fetches the data —
  the gap is processing, not capability. A parallel fetch
  tool would duplicate the existing tool.
- Retroactive check (A-corrective): yes — if parsed output had been
  in context at the decisive step, the agent had enough to answer.

## Candidate C-002
[lens: failure | lever: instruction | intent: corrective]

One-line fix description.

- Tasks affected: task-017, task-033
- Signal: `judge_verdict=format_wrong` on both; reasoning in the
  body is otherwise correct.
- Verified: task-017 step 15 body ends mid-paragraph with no
  answer marker. task-033 step 12 — same shape.
- Why Instruction not Control: the reasoning was correct — the
  agent just didn't know to emit the marker. A mechanical
  post-hook can't extract the answer from narrative prose, and
  there's no tool output to post-process.
- Retroactive check (A-corrective): yes — an explicit format rule
  produces the correct commit path.

## Candidate C-003
[lens: success | lever: instruction | intent: preservative-transfer]

Promote the "emit a `## Plan` block before the first tool call"
habit from an implicit pattern to an explicit template rule.

- Tasks affected:
  - passing (habit fired): task-024, task-055, task-071
  - failing (habit absent at decisive step): task-038, task-093
- Signal: all three passing tasks write a `## Plan` block in step
  1-2 and keep tool calls under 6 (short `tool_call_counts`); the
  two failing tasks skip the plan, fan out to 14-18 tool calls,
  and `exit_reason=budget_exceeded`.
- Verified: task-024 step 1 body quotes the `## Plan` block;
  task-055 step 2 — same shape; task-071 step 1 — same shape.
  task-038 steps 1-3 — no planning block, tool calls thrash on
  adjacent queries; task-093 step 1 — same shape.
- Why Instruction not Control: a Control hook would inject the
  plan mechanically, bypassing the agent's own scoping — the
  planning *content* is what made the passing cluster short, not
  the presence of a header string. An Instruction rule keeps
  the scoping agent-side.
- Retroactive check (C-preservative-transfer): yes — task-038 and
  task-093's thrashing begins before step 4; a planning rule
  applied up-front would have forced query scoping and kept the
  step count under budget.

The "Why X not Y" field is the main failure mode to avoid

A candidate that skips this field almost always picks the lowest-friction lever (usually Instruction — "add a rule to the prompt") on evidence that points at a heavier lever (Action / Control). Writing the "Why X not Y" line forces you to argue against the adjacent alternative, not just in favour of the one you picked. Examples of the mistake this catches:

• Capability-gap signal, Instruction lever, no "Why not Action" → usually means the agent has no way to do the thing, but the author defaulted to a prompt patch because it was easier to write. Re-pick as Action unless the counter-case in "Action vs Instruction" applies and you can write it out.
• Control-shaped signal (tool returns unusable data), Action lever, no "Why not Control" → usually means an on_after_tool normaliser would have been the narrower fix.
• Success habit, Instruction lever, no "Why not Control" → a Control hook mechanically injecting the habit is tempting ("always write a ## Plan block") but bypasses the agent's own scoping; argue why the habit needs to stay agent-authored. Or the opposite: a Control lever with no "Why not Instruction" → you're hard-coding a habit the agent could have learned to apply selectively.

If you can't write the "Why X not Y" in one sentence, the lever choice isn't ready and the candidate needs more investigation.

Concrete, benchmark-specific examples (e.g. answer-format markers, role-play protocols, verifier scripts, domain-specific tool specs) live in the <bench>-playbook skill, not here — this skill is benchmark-agnostic.

Rank candidates by evidence strength × expected leverage. Ship the ones you actually implement — a candidate that isn't implemented shouldn't appear in cited_candidates on the journal entry.

Cross-reference in the journal

When this round's config changes, the latest journal entry's frontmatter must cite at least one of the candidate IDs:

cited_candidates: [C-001, C-002]

See journal skill for the full frontmatter schema. The evidence gate rejects any non-noop round where:

• candidates.md is missing, OR
• it has no ## Candidate C-N sections, OR
• cited_candidates is empty or missing, OR
• a cited ID doesn't have a section in candidates.md.

Noop rounds (cp current_config output/config.yaml) don't need candidates.md — the orchestrator only enforces evidence when the structural diff is non-empty.

Pitfalls

• Jumping to an encoding before the retroactive check. The first plausible story is often downstream of the real pattern — whether you're closing a gap (Variant A) or transferring a habit (Variant C).
• Reading only failures. Shipping fixes while the successes that made the baseline work silently regress is the most common way a round lands net-negative. Sweep passing clusters with the same seriousness as failing ones.
• Confusing judge signal with eval signal. The judge is an opinion; when eval_passed disagrees, believe eval_passed.
• Single-task optimisation. Idiosyncratic observations — failing OR passing — shouldn't shape the config. Log them under NEEDS_FROM_HUMAN in the journal and move on.
• Generic tool replacement. New tools should close a specific class; a "better WebFetch" competes with WebFetch in the general case and loses. Scope to the cluster.
• Layer-by-lens reflex. "Failure → prompt fix", "capability gap → new tool", "success → new processor". All three are often wrong. Always re-check the lever against the evidence.
• Encoding a habit that was already free. A preservative-lock candidate whose Variant-B check returns no (the passing tasks would still pass with the habit removed) is dead weight — it adds config surface area without paying for it. Drop, don't ship "to be safe".