Method Evolution

Method-Evolution — the outer loop (∂L/∂method)

The Metaphor

The blacksmith sharpening the tool. Three botched hinges in a row — the blacksmith doesn't re-hammer the last hinge harder. She examines the hammer. The striking face is dulled from a thousand blows. The tool has drifted, and every future hinge will inherit the drift until the tool is re-tempered. In engineering: when the same rubric violation appears across three distinct tasks, the problem is not in the tasks. The skill itself needs work. Sharpening the tool is slower than hammering harder — but every future task benefits.

Overview

This skill is the third gradient of Gradient Descent for Agents. Siblings: inner-loop skills move θ = code, iterative-refinement moves y = deliverable, dialectical-cot moves t = reasoning chain. This skill moves m = method — the skills, prompts, rubrics themselves. It is the rarest and most load-bearing of the four loops — the only one that can actually improve the bundle over time, and the corrective channel when any of the other three loops systematically drifts (including the thinking-levels scorer, see automatic trigger below).

If your agent keeps making the same kind of mistake across completely different tasks, the mistake is not in the tasks — it's in the method that's instructing the agent. That means a skill is too vague, a rubric is too permissive, a prompt lets a rationalization slip through.

Updating the task fixes this bug. Updating the method fixes the bug and prevents the next 10.

Core principle: method-evolution is SGD on the m-axis (the skills, prompts, rubrics themselves), not the inner-loop code axis, the refinement y-axis, or the CoT t-axis. It is the rarest and most load-bearing of the four loops — and the only one that can actually improve the bundle over time.

See ../../docs/ldd/convergence.md §1 for the four-loop model and §3.4 for the "moving-target loss" anti-pattern that this skill prevents.

When to Use

Invoke only when all of:

• You have observed the same rubric violation / symptom-patch / escalation in ≥ 3 distinct tasks across different domains
• The method's Red Flags list does not already cover this case
• Patching each task individually would not prevent the next occurrence
• You have a task suite (≥ 5 tasks, varied) you can re-run to measure Δloss across the method change

Do not use when:

• You saw the pattern once or twice — that's noise, not signal. Wait for 3 distinct occurrences.
• The pattern is specific to a single task/domain — update the task, not the method.
• You don't have a suite to measure against — changing the method without measurement is drift, not evolution.
• The problem is that the method is verbose or ugly — refactoring style is not evolution. Evolution requires a behavior change backed by measurement.

Automatic trigger: thinking-levels scorer weight adjustment

The thinking-levels scorer in scripts/level_scorer.py is a parameter of the method (see docs/ldd/thinking-levels.md). When the scorer systematically picks levels that are too low — shipping silent symptom-patches where a higher-level pass would have caught the structural origin — method-evolution is the correction channel.

Trigger condition (all must hold):

• Across the last 20 tasks persisted in .ldd/trace.log:
• There are ≥ 3 tasks with dispatch_source == auto-level AND level_chosen ∈ {L0, L1} AND a later task closing in the same code area with loss_final > median(loss) AND regression_followed = true set on the original task.
• The signal pattern across those 3+ tasks is consistent — e.g. all three tasks contained a contract-rule-hit cue that the scorer under-weighted, or all three contained an ambiguous phrase the detector missed.

Proposed change: a single weight adjustment to the signal most correlated with the missed tasks — typically +1 to one of ambiguous / contract-rule-hit / layer-crossings. Boundary changes (widening a bucket) are allowed only when ≥ 3 of the 3+ tasks score exactly on the boundary.

Measurement: run the 9-scenario fixture suite under tests/fixtures/thinking-levels/ against the proposed weights. If mean loss across the suite regresses, apply the standard rollback protocol below (halve the adjustment, re-measure; if still regressing, revert entirely).

Asymmetric-loss note: when choosing WHICH signal to adjust, prefer changes that bias the scorer UPWARD on ambiguous boundary cases. A tuning that catches one new low-side miss at the cost of two new high-side misses (too-high levels on simple tasks) is still a net improvement per the lieber schlau als zu dumm rule — but mean-loss regression on the fixture suite is still disqualifying.

The trigger is auto-detected: any agent with access to .ldd/trace.log can read the last 20 entries, compute the condition, and propose the adjustment. No human needs to spot the pattern for method-evolution to fire.

The Evolution Step

A single method-evolution step has a rigid structure. Treat it as an experiment, not a refactor.

1. Pattern identification

Name the pattern concretely:

"In tasks T1 (root-cause debugging on a contract error), T2 (dialectical recommendation on a retry design), and T3 (refinement of a technical doc), the agent bypassed the [specific rubric item R] via the rationalization [specific phrase]. The skill's Red Flags list does not currently cover this phrase."

No hand-waving. The pattern must be specific enough that a reader can grep for its occurrence in future runs.

2. Method-change proposal

Propose one change. Not three. One.

If you can't name which line of the method-artifact changes and which rubric item it enforces, you don't have a proposal — you have an urge.

3. Measurement (required)

Compute Δloss_method across the task suite:

Δloss_method = (Σ_t loss(t | method_before)) / |suite|  −  (Σ_t loss(t | method_after)) / |suite|

This requires:

• The task suite fixtures (stored, reusable)
• The original method-artifact version (git archive before edit)
• The edited method-artifact
• A way to re-run each task under each method and score against rubrics

On Claude Code, scripts/evolve-skill.sh (optional tooling) can automate the rerun. On other platforms, run the suite manually.

Acceptance criteria:

• Δloss_method > 0 on the task that motivated the change
• Δloss_method ≥ 0 on every other task in the suite (no regressions)
• Otherwise roll back and halve the learning rate (make a smaller change and re-try)

4. Commit

The commit for a method-evolution step has a specific shape:

evolve(skill-name): <one-line pattern>

Observed pattern across tasks T1, T2, T3:
  <the rationalization / violation, verbatim>

Change: <what line changed in the skill-artifact>

Measured Δloss_method across suite of N tasks: +X.Y
Regressions: none / [task T4: +0.Z, rolled back]

Suite fixtures: tests/fixtures/*/

No anonymous "improve skill" commits. Each evolution step is auditable and reversible.

Red Flags — STOP, this isn't evolution

• "I want to make the skill clearer" → refactoring prose, not evolving behavior
• "The skill feels long, let me trim it" → style, not evolution; don't pretend otherwise
• "I'll update the rubric to match what the agent did" → that's moving-target loss (see docs/ldd/convergence.md §3.4), forbidden
• "We saw this pattern once last week" → not enough signal; wait for 3 occurrences
• "I don't need to measure, the change is obviously better" → exactly the symptom this skill prevents in code; same rule applies to the method itself
• Editing the skill without running the suite → your edit is a random direction in method-space

Rollback discipline

Rollback is not a failure state — it is the most common outcome of a method-evolution step. Most proposed changes don't improve mean loss across the suite; they improve one task and regress another.

• If Δloss_method ≤ 0 on the motivating task → the change doesn't work; revert.
• If Δloss_method > 0 on motivating task but a regression appears on another → revert, then propose a narrower change.
• Track every rollback. The pattern of rollbacks across evolution attempts is itself a signal — if five proposals all rolled back, the skill may be at a local optimum and you need iterative-refinement on the skill itself (refinement of its prose), not further behavior evolution.

Anti-Patterns

Relation to other skills

• loop-driven-engineering dispatches this skill when the same failure recurs — "same gate rejecting with the same reason ≥ 2 iterations → stagnation" is an inner-loop signal; "same stagnation pattern across 3 tasks" is an outer-loop signal.
• iterative-refinement is the y-axis counterpart — refinement polishes a deliverable; method-evolution evolves the skills that produced deliverables.
• drift-detection feeds this skill — patterns found by drift scans are candidates for evolution steps.
• dialectical-reasoning runs over every proposed evolution step: thesis (proposed change), antithesis (what it might regress on, who would disagree), synthesis (narrower change or replaced).

How to Apply — checklist

1. Collect evidence. Three distinct tasks, same pattern. Cite them with commits/artifacts.
2. Name the specific pattern. One sentence, ideally greppable.
3. Propose one change. Name the file, name the line, name the rubric item.
4. Measure. Run the suite under method_before and method_after. Record Δloss_method.
5. Decide. Δloss > 0 no regressions → commit with the canonical message shape. Otherwise → roll back, halve learning rate, retry with a narrower change.
6. Log to outer-loop history. Keep a record of evolution attempts — accepted and rolled-back. Future evolution decisions depend on this history.

Common Rationalizations

Real-World Cue

A healthy method-evolution history is mostly failed experiments. If every evolution step you logged was accepted without rollback, you're not being rigorous — you're either measuring wrong, skipping measurements, or only proposing trivial changes. A bundle like LDD should show ~1 accepted evolution step per 3–5 attempts.