Prompt Retro

prompt-retro

The meta-loop: the setup studies its own usage and proposes its next improvement. Born from a full-history audit (thousands of human prompts) that found rituals were being codified at rep ~25 instead of rep ~3. This skill makes that audit cheap and recurring, so the gap shrinks to one retro cycle.

State

In this skill dir:

• retro-log.md — one section per run: date, window, lens, metrics, proposals. Read it first; the current run's window starts where the last one ended.
• briefs/YYYY-MM-DD.md — the FULL brief of every past run, exactly as delivered. Read the last 2-3 briefs before writing anything: they are the record of what the user has already been told. A finding repeated verbatim across briefs is noise; either drop it, or escalate it explicitly ("third retro in a row — this isn't getting fixed, here's why I think that is"). Save the current run's brief here before surfacing it.

Protocol

1. Extract. python3 "$SKILL_DIR/scripts/extract_prompts.py" <scratch-dir>/digests --since <last-run-date> — walks every ~/.claude* config dir, keeps only human-typed prompts (drops task-notifications, hook/judge prompts, harness runs), writes one digest per project with session headers (interrupts, slash-commands) and timestamped prompts.
2. Fan out analysis agents (one per major project digest + one quantitative cross-cutting pass), each reporting:
   • recurring task types prompted by hand (with counts + verbatim examples)
   • friction: corrections ("no,/wrong/still/why did you"), interrupts, re-sends, stalls
   • context/boilerplate retyped across sessions
   • rep-2 violations: any instruction/doctrine typed ≥2 times → immediate codification candidate
   • go-signal tax: bare "continue/lets roll" counts (loops that should be /loop or background agents)
   • secrets pasted in chat (flag for vault + rotation) The quantitative pass: slash-command league table, correction rate per project, prompt prefixes, nudge counts.
3. Rotate the lens. Each run ALSO takes one fresh angle the previous runs didn't — pick the next from this wheel (recorded per run in retro-log.md) or invent a better one: delegation efficiency → verification & rework loops → comms quality → security hygiene → skill adoption (are built skills firing? repetition decay per skill) → session structure (stalls, compactions, parallel-chat collisions) → model/cost usage → what-changed-this-window. The standing metrics stay every run; the lens is where new findings come from. Once the wheel has been fully walked, inventing a new lens is REQUIRED, not optional — good sources: a lens suggested by last run's anomalies, a mental-model concept applied to the workflow, or the inverse of a previous lens (e.g. "where did Claude over-verify" after "under-verify"). 3b. Freshness gate. Diff your draft findings against the last 2-3 briefs (see State). Every finding gets one label: NEW (this window), PERSISTING (said before — include only with a why-it-persists analysis), or RESOLVED (celebrate it: proof the loop works). A brief that is mostly PERSISTING items without escalation is a failed run — dig for what's new instead.
4. Synthesize into a ranked brief: (a) skill/hook/setup candidates by frequency × pain, each with verbatim evidence; (b) habit deltas vs the last retro (did correction rate drop? nudge tax? did proposed skills get built and used?); (c) the lens findings; (d) the 2-3 things to build this cycle.
5. Close the loop: append the run to retro-log.md (date, window, lens used, metrics, proposals).
6. Surface it: deliver the brief however the user prefers (email tooling if wired up, subject "Prompt Retro YYYY-MM-DD — "). Otherwise write it to ~/Desktop/prompt-retro-YYYY-MM-DD.md and fire a macOS notification: osascript -e 'display notification "retro brief on Desktop" with title "Prompt Retro"'.

Judging improvement between runs

The score that matters is repetition decay: a ritual flagged last retro should appear near-zero this retro (because a skill absorbed it). Rituals that persist across two retros despite a built skill mean the skill isn't triggering — audit its description, don't build another one.

Baseline

Your first full audit sets the baseline: correction rate per project, go-signal tax (share of bare "continue" prompts), and share of raw transcript "user" turns that are machine traffic. Record it in retro-log.md and beat those numbers on subsequent runs.