majiayu000 Run the Kai learning retrospective. This is how the harness gets smarter: raw failure logs become lessons, repeated lessons become enforced checks. Read memory/MEMORY.md first for the graduation ladder.
When to run
- Monthly, or after any sprint that produced 5+ gated pieces
- Whenever the same gate failure shows up twice in one session
- When a 30-day performance check grades new losers
Step 1 — Mine the gate logs
python scripts/self_improvement/lesson_capture.py mine
This groups recurring failure signatures from data/learning/gate_runs.jsonl. Append candidates with --write. If the log is empty, note it and move on — the gates only log when they run.
Step 2 — Diagnose losers
python scripts/self_improvement/lesson_capture.py losers
For each undiagnosed loser, read the piece and its content_log.json entry, write a one-line diagnosis (hook type, persona mismatch, seasonality, thin proof — name the cause, not the symptom), and add it to memory/what-doesnt-work.md under "Measured losers" with the piece id. Check seasonality and competitor moves before blaming the content (see memory/edge-cases.md EC-15).
Step 3 — Triage every lesson
Go through memory/lessons.md:
| Verdict | Criteria | Action |
|---|---|---|
| Promote | Fired 3+ times, or checkable by a regex/threshold | Graduate it (Step 4), mark (promoted) |
| Keep | True, useful, not yet recurring | Upgrade candidate → active if verified |
| Merge | Near-duplicate of another lesson | Combine into the more general one |
| Retire | No longer true (platform changed, gate fixed) | Mark (retired) with the reason — never delete |
Step 4 — Graduate promoted lessons
Pick the strongest enforcement target, in this order:
- Lint rule / contract check — new entry in a banned-word tier, a new overclaim regex in
scripts/quality_gates/seo_lint.py, or adeterministic_checksline in the format's skill contract. - Checklist line — the relevant
knowledge/checklists/*.md. - CLAUDE.md / framework rule — only for judgment calls code can't check.
Non-negotiable: any change to a gate script requires a matching case in evals/golden/manifest.json (one sample proving the new check fires, and confirm the existing pass samples still pass), then:
python scripts/quality_gates/golden_check.py
A gate change without a golden case is not a promotion — it's a regression waiting to happen.
Step 5 — Refresh the memory index
- Update the "Current standing lessons" section of
memory/MEMORY.md(keep the file under 200 lines). - Cross-check
memory/edge-cases.md: mark any entry whoseEnforcement: noneyou just fixed; add new edge cases discovered this cycle.
Step 6 — Report
Output a retro summary:
## Kai Retro — [date]
**Mined:** [N] recurring failure signatures ([gate]: [signature] ×[count], ...)
**Losers diagnosed:** [N] ([id]: [one-line diagnosis], ...)
**Promoted:** [lesson] → [enforcement target] (+ golden case [id])
**Retired:** [lesson] — [reason]
**Edge cases:** [new/closed entries]
**Open risks:** [lessons at 2 occurrences — one more and they must promote]
Commit the memory and gate changes together so the diff shows the lesson and its enforcement side by side. If a promotion changes publishing behavior (new hard block), flag it for human approval rather than applying silently.