Learning Loop

ABOUTME: Cross-repo learning loop, turns scattered LEARNING.md into ranked harness changes

ABOUTME: Deterministic ingest (script) + recurrence detection (agent) + change-contract output

Learning Loop

Turns the LEARNING.md retrospectives scattered across every repo into process improvements. The value is not the count of lessons, it is the recurrence: a failure shape that appears in one repo is an anecdote, the same shape across repos is a signal worth a mechanical fix.

Two halves, by design:

1. Ingest (deterministic, no LLM). scripts/learning_corpus.py discovers all LEARNING.md, dedupes working copies, splits each into atomic learnings, emits a JSONL corpus. Reproducible and free.
2. Recurrence (agent pass). An agent clusters the corpus by failure shape, keeps only patterns spanning two or more distinct repos, ranks them, and proposes one harness action per pattern with a six-field change-contract.

This pairs with, but is distinct from, related tools. learning-docs writes a single project's LEARNING.md (the input to this loop). harness-mechanic reads execution traces and token baselines (mechanical signals: cost, tool-call shape); this loop reads the human retrospectives (what actually went wrong and why). knowledge-sync promotes vault patterns to skills; this loop promotes cross-repo failure-modes to harness changes.

When to run

On a human schedule (monthly, or after a milestone closes across several repos), never autonomously. The corpus is cheap to rebuild; the agent pass costs tokens, so do not loop it.

Cadence guard

Process

Step 1: Build the corpus (deterministic)

cd <claude-forge>
make learning-corpus            # writes quality_reports/learning_corpus/corpus.jsonl
# or, to scan a different root or see coverage:
uv run scripts/learning_corpus.py --root ~/Development --stats

The output lives under quality_reports/learning_corpus/, which is gitignored: it contains private war stories from work repos and must not be committed.

Step 2: Recurrence detection (agent)

Spawn one analysis agent over the corpus. Give it, verbatim, the rules that keep counting honest:

• Logical repo grouping. The repo field is a path; collapse same-repo variants (e.g. a backbone present at two paths is one repo) before counting. Group products: all hikmaAI/* is one product, all Wishew/* is one product, each private/* is its own side-project. Cross-product recurrence is the strongest signal.
• Threshold. Keep only clusters with two or more distinct logical repos. List singletons separately as a watch-list.
• Rank by (recurrence count) times (cross-product breadth) times (preventability by a mechanical change).
• Cluster by failure shape, not surface topic. "Everything reports success but nothing verifies it" is a shape; "a Redis bug" is a topic.

For each ranked pattern the agent outputs: a name, the failure shape in one sentence, the member learnings as evidence (repo, date, title), a breadth verdict, exactly one proposed harness action (new hook / new or edited rule / skill update / review-checklist item, concrete), and a six-field change-contract (see rules/harness-changes.md and the template at quality_reports/harness_changes/TEMPLATE.md).

Write the report to quality_reports/learning_corpus/recurrence-report.md. Demand skepticism: better five real patterns than fifteen forced ones.

Step 3: Triage with the human

Present the executive summary and the ranked patterns. For each one the human accepts:

1. Copy the agent's change-contract to quality_reports/harness_changes/YYYY-MM-DD_<slug>.md.
2. Implement the single action (one hook, one rule edit, one skill update, or one checklist line).
3. Land the contract with the change, per rules/harness-changes.md.

Prefer the cheapest fully-mechanical win first (zero-judgment, near-zero false-positive hooks) over the highest-value-but-fuzzy one. A noisy hook trains the human to ignore it.

Step 4 (optional): Archive

Archive the report to the vault for trend tracking across runs, so the repeat-pattern rate (how many of last run's patterns recurred) becomes visible over time.

Doc-gardening pass

The rot this loop fights in process shows up in the governance docs themselves: rules pointing at renamed files, the skills index routing to skills that no longer exist, SKILL.md claims contradicted by newer change contracts. OpenAI's harness-engineering team runs a recurring "doc-gardening" agent for exactly this. Run it on the same human cadence, or after a batch of harness changes lands.

1. Deterministic scan. make doc-garden (wraps scripts/doc_gardening.py): reports dead backticked repo paths across CLAUDE.md.example, README, rules/, agents/, skills/*/SKILL.md, plus skill-index drift. Exit 1 on findings. UNLISTED-SKILL lines are advisory input for step 2, not defects (skills/_INDEX.md is a curated map: a skill absent from it still auto-triggers from its own description).
2. Agent pass (judgment). Give one agent the scan output plus the most recent change contracts (last 5, ls -t quality_reports/harness_changes/) and ask, per governance doc: does any claim contradict what the code, hooks, or a newer contract now says? Output fix-up proposals, one line each: file, stale claim, what changed, proposed edit.
3. Triage. Trivial fixes (dead path, renamed file) apply directly. Anything touching a trigger surface (SKILL.md description, rules/) gets its own change contract per rules/harness-changes.md.

Anti-goals for this pass: no prose rewrites, no reorganization, staleness only.

Output contract

• corpus.jsonl: one atomic learning per line, fields {repo, source, section, date, title, body, body_lines}.
• recurrence-report.md: executive summary, ranked patterns with evidence and change-contracts, singleton watch-list, data caveats.
• Zero to N change-contracts under quality_reports/harness_changes/, one per accepted pattern.

Anti-goals

• Do not run autonomously or on every session: it is a periodic review, not a hook.
• Do not propose more than one failure mode per change-contract: ambiguous falsification kills the loop.
• Do not commit the corpus or the report: they hold private incident detail.
• Do not treat trace metrics as input here: that is harness-mechanic's job, and the highest-value signals (a green test that lies) are invisible to traces.