Improve System

Improve System — the measured improvement loop

Mine the system's own correction history for recurring failure patterns, trend the rejection-rate metric, and propose targeted edits. Propose, never auto-apply. This is the auto-research loop adapted for non-measurable work: chat-history corrections are the proxy signal, qa-rejection rate is the trend metric (video-insights improvement #5, 2026-06-12).

Scope boundary

• /system-audit finds specific bugs and drift to fix once. /improve-system finds recurring behavioural patterns across runs and turns them into standing rules.
• /li curate consolidates agent overlays into skill diffs (3+ run threshold). Do not duplicate it: when a pattern's evidence lives only in overlays, note "route via CURATE" instead of drafting the diff here. This skill's distinct ground is the QA logs, the rejection-rate metric, and session-history corrections.
• /grade-system grades overall quality posture. This skill measures corrections and spend; it grades nothing.
• Model-routing questions are not adjudicated here from logs alone — see "Routing questions" below.
• An improvement loop without a metric is a ritual. Every run reports the metric, even when it proposes nothing.

Steps

1. Gather signals (work folder, read in parallel):

   • agent-improvements/qa-rejection-log.md — verdicts, re-delegation counts, reasons (primary metric).
   • agent-improvements/qa-disagreement-log.md — where Ane or Ann overrode qa verdicts.
   • agent-improvements/cost-calibration-log.md — estimate-vs-actual drift, AND the numbered calibration-signal list inside each session row, which is the richest correction channel the system has. Every row carries 5 to 9 numbered signals naming what went wrong, what was misdiagnosed, and what was fixed at source. Read the signal lists of every row appended since the last run, not just the cost columns. Run 5 (2026-07-31) evidence: the three listed direct-mode channels returned zero while 19 signals across 3 rows carried every pattern the run found. The table is the file's LAST section (restructured 2026-08-05, Run 7 P4, so end-of-file appends land inside it); the newest rows are the table's tail.
   • ## Active Improvements sections of agent-improvements/{ann,vi,li,researcher}-overlay.md.
   • .remember/recent.md and .remember/today-*.md — session-history proxy for corrections Ane made in conversation.
   • agent-improvements/pre-flight/*-lessons-learned.md — the direct-mode correction signal for the four classic deliverable classes. Most work is now solo/direct (no Ann/Vi orchestration), so the qa-rejection-log is starved by design; these logs are where direct deliverable corrections land. Read every *-lessons-learned.md under agent-improvements/pre-flight/.
   • Feedback memories created or updated in the window — the direct-mode signal for everything OUTSIDE the four class logs. Corrections to non-classic deliverables (Excel tools, ops docs, pilot plans, emails, skills) land as feedback_* memories in the session memory directory, not in a class log. List files by mtime (ls -l --time-style=+%Y-%m-%d on the memory dir) and read the in-window feedback_* entries. Run 2 (2026-07-17) evidence: 10 weeks of zero class-log entries while 3 real corrections sat in feedback memories.
   • agent-improvements/improve-system-runs.md — prior runs (avoid re-proposing rejected items).

2. Trend the metric (four signals).

   • Orchestrated runs (qa-rejection-log). Report: total orchestrated runs logged, share with re-delegations or non-PASS verdicts, top 3 recurring reasons, direction vs the previous run's figures. This log fuels ONLY on Ann/Vi orchestration. If it is empty after sessions that clearly ran orchestration, the finding is "the wrap-up 4b logging step is failing" — a logging gap. But if the window was direct-mode work, an empty log is expected, not broken — verify which before concluding (check the session history and cost-calibration-log for orchestrated runs in the window; confirm the log's creation date predates the runs you expected).

   • Direct-mode corrections, classic classes (pre-flight lessons-learned logs). For each *-lessons-learned.md, report the count of entries in the window and the recurring correction themes. These four logs (ToC, indicator framework, evaluation design, evidence brief) only populate when those deliverable classes actually ship — empty logs during a window with no such deliverables are expected, not broken. Trend it the same way: rising repeat-corrections of the same theme = a rule not being applied.

   • Direct-mode corrections, everything else (feedback memories). Report the count of feedback_* memories created or updated in the window and their themes. When the window's work mix falls outside the four class logs, this is the live direct-mode signal. Repeat themes across memories = a candidate pattern; a correction that matches an EXISTING feedback memory's rule = a rule-not-applied failure, weight it heavier.

   • Direct-mode corrections, calibration signals (cost-calibration-log rows). Report the count of rows appended since the last run and the count of numbered signals across them, then cluster the signals by theme. Weight a signal that names a rule which already exists as a rule-not-applied failure, exactly as for feedback memories. This channel does not go quiet when the work mix shifts, because /wrap-up writes a row for every session, which is the property the other three lack.

3. Cluster patterns. A pattern needs 2+ independent occurrences across runs or logs. One occurrence is a watch item, listed but not actioned. Name each pattern in one sentence with its occurrences cited.

4. Propose edits. Per pattern: target surface (CLAUDE.md rule, skill edit, agent edit, checklist, feedback memory, or PreToolUse hook for never-do rules), draft wording, expected effect on the metric. Present all proposals as a numbered list and wait for Ane's per-item approve/reject/defer.

5. Apply approved items only. Use the Edit tool, scope-bounded. Apply mel_wiki/wiki/concepts/edit-preservation-protocol.md when target file exists.

   Edit the canonical file, never a generated mirror. Several targets this skill writes to are mirrors that a sync script regenerates: claude-ai-shareable-export/* is generated from the repo-root file of the same name, .claude/skills/<name>/SKILL.md from the personal-skills clone, .claude/agents/* from ~/.claude/agents/. An edit to the mirror survives until the next sync, then vanishes with no error. Before editing any file under an export, mirror, or cache directory, find what generates it (grep the sync scripts for the filename) and edit that instead. Evidence: Run 4 wrote the claude.ai scope-gate change into claude-ai-shareable-export/ and sync_shareable_export.py overwrote it in the same turn.

   When an approved item edits a skill or agent file, re-run the harness before closing the run. Line budgets in SKILL_BUDGETS (tests/run_tests.py) gate those files, so adding a rule can breach one and the failure surfaces only at the final harness run. Read the current budget from that dict rather than assuming it; the figures move. Fix a breach by folding the addition into an adjacent paragraph, never by trimming content you did not write. Evidence: Run 3's vi one-retry rule took the file two lines over and cost a follow-up commit.

6. Close the run. Append a dated summary (metric figures, patterns found, proposals + Ane's decisions) to agent-improvements/improve-system-runs.md (create with a # Improve-System Run Log header if absent). Then mark the cadence: python ~/.claude/hooks/maintenance_due.py --mark improve-system 2>/dev/null || true.

Spend-audit mode — where the tokens actually go

Runs on the spend triggers ("where are my tokens going", "token audit", "why was that session expensive", "context optimisation"), or when step 2's trending surfaces a cost anomaly worth explaining. Read-only: the output is a Tier 1 working brief; the only permitted write is an opt-in backlog line. Folded from the token-economy-audit backlog entry (2026-08-05); the 2026-07-30 hand-run pass is its evidence base.

1. Compute the always-loaded floor in-session; never quote a stored figure. Sum the live sizes of user + project CLAUDE.md, MEMORY.md, skill frontmatter total, agent frontmatter total, and SessionStart injections. The computed-counts rule applies hardest here: a spend figure nobody computed this turn is a ⚠️ data gap, not a number.
2. Mine agent-improvements/cost-calibration-log.md as a dataset, via ctx_execute_file — never Read. The file exceeds 140k chars and the large_read_gate hook refuses an unbounded Read. Extract per-row $/1k-output, cache-write ratios, and every "Calibration signals" block into a driver table. The three known drivers (output volume, cache write, spec-then-clear) were only ever found by reading the log end to end; the audit exists so nobody repeats that by hand.
3. Rank levers by (saving × confidence) / effort. Label each lever quality-neutral, quality-positive, or a trade. Never propose a trade without naming exactly what is being traded.
4. What not to optimise — standing rule. Never trim standing instructions to save tokens. The floor is not the cost: cache read runs 93–99%, so holding it costs cents, and the rules inside it are what hold output quality. Spend tracks words authored plus new bytes entering context; the lever is cutting rework, never rigour. A hand-run audit would very likely start by trimming CLAUDE.md, which is the one change that trades quality for pennies — this rule is why the mode exists.

Routing questions — run a pilot, never adjudicate from logs

When the mining surfaces a model-routing question — a correction pattern implicating a model tier, a Haiku-tier candidate, an Opus escalation rule in doubt, a reviewer-model comparison — do not settle it from correction logs or cost rows. Run the seeded-defect blind pilot per mel_wiki/wiki/concepts/routing-pilot-method.md: sealed answer key, ~10 planted defects across gate classes, pre-declared pass criteria, unnamed async Agent spawns so per-arm token usage is observable. The registry (agent-improvements/agent_registry.md § Routing tiers) requires a logged pilot before any Haiku-tier addition, and an escalation removed by a pilot verdict needs a re-probe pilot before reinstatement. A pilot's verdict lands as a PILOT row in cost-calibration-log.md plus the registry edit it implies — propose that edit through step 4 like any other.

Red flags — stop if you catch yourself doing these

• Applying any edit before Ane approved that specific item. "She'll obviously approve" is not approval.
• Promoting a single occurrence to a pattern because it "feels familiar".
• Re-proposing an item the runs log shows Ane rejected, without new evidence.
• Reporting "system healthy" without printing the metric figures.
• Quoting a floor, spend, or cost figure that was not computed this turn (spend-audit mode).
• Settling a model-routing question from logs instead of a seeded-defect pilot.