# Pattern Retrospective > Apply rigorous pattern-retrospective analysis to Claude transcripts. Fires on any study, blog post, handoff doc, retrospective, spec, or improvement proposal that draws conclusions from past Claude sessions. Enforces target-system audit BEFORE specifying its requirements, streaming JSONL parsing, 5-tuple extraction with provenance tags, self-falsification of every cited claim, Bayesian confidence smoothing, and correlation-not-causation discipline. Prevents the credulous-handoff failure mode where ~60% of proposed requirements already exist in the target system. - Skill: `bryceewatson/pattern-retrospective` (Agent Skill, multi-file: 12 files) - Install (CLI): `npx skillmds@latest add bryceewatson/pattern-retrospective` - Raw SKILL.md: https://api.skillmd.com/api/skills/bryceewatson/pattern-retrospective/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: BryceEWatson (https://skillmd.com/u/bryceewatson) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/bryceewatson/pattern-retrospective --- # Pattern-retrospective workflow You're producing a study / blog post / handoff / spec / retrospective that draws conclusions from Bryce's past Claude sessions. Apply this rigor before publishing. The chat-arch v2 Rev 3 plan describes the substrate that will eventually automate this. Until that ships (the chat-arch repo's `_planning/chat-arch-v2-rev3-plan.md`), you are the curator, falsifier, and substrate. The outputs you write become the seed data when chat-arch's automation lands — write them so they're machine-readable later. ## §1 — The load-bearing meta-discipline **Before specifying any external system's requirements, audit that system first.** If you're writing a "handoff to X" / "spec for X" / "improvements to X" / "what X should do" document, spend 30 minutes reading X's code, locked spec, and CLAUDE.md before writing. Lead the document with a "Context: what already exists in X" section. If you can't write that section, you can't write the document. This applies recursively. When you cite a fact about an external system, verify it the same way you'd verify any other agent's cited claim. "X already does Y" is a falsifiable claim — falsify it before relying on it. **Why this rule exists:** the May 2026 handoff to chat-arch failed this check. Roughly 60% of its proposed "chat-arch requirements" already existed in the chat-arch codebase — Phase 1's log-location discovery, Phase 3's false-positive filter, Phase 6's external verification, Phase 7's aggregate metrics, and the "MVP pattern-retrospective view" overlap with the Playbook page. None had been audited. See the chat-arch repo's `_planning/handoff-feedback-2026-05-22.md` for the full critique. Don't repeat that. ## §2 — Storage map (always search both corpora) Bryce's transcripts live in two corpora. Searching only one misses ~10× the data. **Corpus 1 — Claude Code CLI:** - `~/.claude/projects//.jsonl` (main sessions) - `~/.claude/projects///subagents/agent-*.jsonl` (subagent transcripts — sibling to parent, NOT at the same level) **Corpus 2 — Cowork (Claude Desktop local-agent-mode):** - `%AppData%\Claude\local-agent-mode-sessions\\\\agent\local_ditto_\audit.jsonl` (top-level audit) - `/audit.jsonl` (per-cwd audit — PRIMARY user-prompt source) - `/.claude/projects/-sessions-/.jsonl` (CLI subprocess sessions) - `/.claude/projects//.jsonl` (output queue — CANONICAL source for typed text via `queue-operation enqueue` events) - `/.claude/projects/.../subagents/agent-*.jsonl` (subagent transcripts) macOS path swap: `~/Library/Application Support/Claude/...`. The Cowork directory is migrating to `claude-code-sessions`; both names may coexist. ## §3 — JSONL line-shape filter (keep 3, drop 2) Five shapes appear in `type:"user"` lines. Misclassifying the wrong ones is the primary source of false positives. ```text KEEP — typed user prompt (CLI shape, content is an array): {type:"user", message:{role:"user", content:[{type:"text", text:"..."}]}, timestamp} KEEP — typed user prompt (Cowork audit shape, content is a string): {type:"user", message:{role:"user", content:"..."}, _audit_timestamp} KEEP — queue-enqueue event (Cowork output, CANONICAL): {type:"queue-operation", operation:"enqueue", content:"...", timestamp} DROP — tool result wrapped as user (assistant's tool output): {type:"user", message:{role:"user", content:[{tool_use_id:"...", type:"tool_result"}]}} DROP — task-notification wrapper (subagent task descriptor, NOT user-typed): {type:"user", message:{role:"user", content:[{type:"text", text:"..."}]}} ``` Also drop: text starting ``; cat-n format output (`\n 1\t...`); top-level `type:"attachment"`; `type:"system"`; `isSidechain:true`; `isApiErrorMessage:true`; subagent transcript root files (first line is the task description TO the subagent, not user text). Without this filter, the term "adversarial" returned 217 hits across Cowork; with it, ~71 real user-typed deploys. Both numbers are correct — they're counting different things. Use the right one. ## §4 — Streaming parsers, never whole-file loads Cowork transcripts can hit 18MB; subagent files run 250KB × 5–40 per session. Loading whole files into LLM context will fail (this is what broke 11 of 12 sessions in the May 2026 study). Read line-by-line. Capture only: - the user prompt by substring match - each subagent file's task description (first event) and final assistant message - follow-up user turns for validation A streaming extractor produces a complete 5-tuple per session in seconds at any file size. Reference Python: `extract_cowork_5tuple_v2.py` in `scripts/.tmp/adv-study/` if one was committed there. Critically: **this is an LLM-context-window problem, not a Node-heap problem.** Don't conflate the two when writing about it for other systems. ## §5 — 5-tuple extraction with provenance For each pattern invocation: ```text { dispatch: { promptText, sessionId, timestamp, cwd }, findings: [{ sourceSubagentId, findingNumber, description, severity?, domain? }], validations: [{ findingNumber, verdict: confirmed|rejected|ambiguous|no_validation, userMessageRef?, inferredFromAction? }], actions: [{ type: commit|edit|pr_comment|ignored, targetFinding, detail, timestamp? }], outcome: { state: converged|abandoned|hit_cap|continued, iterationsRun, finalUserSentiment?, shippedArtifact? } } ``` Tag every field with provenance at write-time: - `deterministic` — parsing-rule extraction (timestamps, sessionIds, action types, commit hashes from tool calls) - `llm-derived` — read-and-classify (description, severity, sentiment, validation verdict) - `inferred-from-action` — implicit (user moved on without explicit yes/no; treat as weak signal) - `falsifier-verified` — passed an independent verification pass against ground truth ## §6 — Falsify your own claims before publishing This is the discipline that distinguishes a real retrospective from a credulous one. | Claim type | Verification | |---|---| | Commit hash | `git -C log --oneline \| grep ` — exists or not | | PR number | `gh pr view ` — exists, state matches claim | | URL | WebFetch or `curl -s ` — page exists, content matches the citation | | File:line reference | Read the file — line says what's claimed | | External fact | Web search or fetch authoritative source | | Math claim | Recompute from source numbers | | **Claim about external system** | **Read the system's spec, CLAUDE.md, key source files. Don't speculate.** | The May 2026 adversarial-subagent study found: - 1 fabricated arXiv authorship (TRUE — verified externally; real authors were de Groot, Aliannejadi, Haas) - 2 hallucinated commit hashes (`1e00c87`, `b91ced` — neither in repo, dropped) - 1 wrong mean (claimed 1.8, recomputed as 2.18 from raw counts) Without falsification, all four would have shipped as fact. **Apply the same rigor to your own document.** "X already does Y" is the most-skipped category — the May handoff cited several "chat-arch already does Z" claims that weren't checked. They turned out to be wrong about the existing state of chat-arch. ## §7 — Bayesian confidence smoothing When summarizing patterns across sessions: ```text confidence = supporting / (supporting + contradicting + 2) ``` | Evidence | Confidence | Interpretation | |---|---|---| | 1 supporting / 0 contradicting | 0.33 | Candidate — NOT "always" | | 2 / 0 | 0.50 | Pattern emerging | | 6 / 0 | 0.75 | Validated — eligible for proposed-rule status | | 6 / 1 | 0.67 | Validated stays just below the promote threshold | | 12 / 1 | 0.80 | Strong evidence | Never write "X always happens" from a single observation. The +2 smoothing prevents the overclaim. Match writeup confidence to the evidence smoothing. ## §8 — Correlation, not causation When a pattern correlates with good outcomes: - Say "correlates with" — never "causes" - Show SE alongside the difference when the cited-sample is small (n<5) - Don't rank patterns by correlation when the underlying sample is too thin to be significant - Use a permutation test or Welch's t for any "this pattern → better outcomes" claim - Disclose the test in the methodology paragraph ## §9 — Writeup structure (lifted from chat-arch's Playbook surface) Every retrospective document carries: 1. **Methodology paragraph at top** — corpora searched, sample sizes, extraction protocol, key caveats. One paragraph; primes the reader's epistemics. 2. **Synoptic counters** — 4 KPI numbers (invocation count, sessions covered, falsification stats, evidence quality). Synoptic-before-detail. 3. **Ranked findings with explicit `#1`, `#2`** — rank is a claim worth showing, not implicit. 4. **Per-claim provenance tags inline** — `[deterministic]` / `[llm-derived]` / `[falsifier-verified]` / `[user-confirmed]` near every load-bearing number or claim. 5. **Significance gates** — display correlation only when |Δ|/SE > stated threshold. 6. **Explicit non-claims** — a "what this data does NOT support" section, distinct from the limitations. 7. **Limitations section** — sample size, selection bias, time window, who/what was excluded. 8. **Next experiments** — what would close each non-claim. Progressive disclosure for high-count example lists: "Show 3 of 61 examples" pattern. Don't flood the page. ## §10 — Quality gate before publishing Don't ship the document until every answer is "yes": 1. Have I audited every external system I make claims about? 2. Have I falsified every cited commit / URL / file-ref / number / system-claim? 3. Have I applied confidence smoothing to summary claims? 4. Have I tagged every claim with its provenance? 5. Have I disclosed correlation-not-causation where relevant? 6. Have I included a "what the data doesn't support" section? 7. Have I verified that nothing I'm proposing already exists in the target system? 8. Would a falsifier-agent reading just my document be able to refute any specific claim with a Read or git or web-fetch? If any answer is "no," the document isn't ready. The recursive falsifier-on-self is item 7 — it's the discipline that would have caught the May handoff. ## §11 — Output location Save outputs into structured locations so they're addressable when chat-arch's automation lands: - **Retrospectives / studies** → `research/studies/YYYY-MM-DD_.md` - **Handoffs / specs** → `_planning/.md` (in the repo of the target system, not this one) - **5-tuple extractions** → `research/studies/_data//5-tuples/_.json` - **Inventory tables** → `research/studies/_data//inventory.csv` or `.json` When chat-arch v2 Rev 3 ships, these become seed data. Keep them machine-readable. ## §12 — Self-check: am I being credulous right now? If a document feels like it's writing itself faster than you can verify its claims, you're probably being credulous. Symptoms: - You haven't opened a single file in the target system in the past hour - You're enumerating "phases" for another system without having read its existing phases - You're citing version numbers, commit hashes, or file paths without checking - The document is feeling clean and confident at a suspicious clip - You're describing failure modes you encountered as if they're the target system's required features Slow down. Run §6 falsifier on your own draft. Audit any external systems mentioned in §1. Re-check the math. Confident-and-clean is not the same as correct. ## §13 — Connection to chat-arch v2 Rev 3 This workflow is what chat-arch v2 Rev 3 (the chat-arch repo's `_planning/chat-arch-v2-rev3-plan.md`) will eventually automate. Until that ships: - You are the **curator** — deciding what's worth retrospecting - You are the **falsifier** — verifying citations - You are the **substrate** — your written outputs are the durable record When chat-arch v2 Rev 3 lands, the curator-agent + falsifier-agent + provenance-tagged Narrative entity will absorb these responsibilities. Until then, the discipline is yours to enforce manually. Write outputs in a shape that maps cleanly to the future substrate (the 5-tuple structure in §5 was designed for this). ## §14 — Substep contracts (the "stop reinventing" answer): - **For corpus enumeration + line-shape filtering:** use the `lib/cowork_filter.py` helper (shipped with this skill). Mirrors `chat-history-search`'s §3 filter rules; iterates Cowork + CLI corpora; yields parseable prompts. (`chat-history-search` itself currently outputs human-readable markdown only; the helper is the programmatic-consumption answer until chat-history-search adds `--output-format jsonl` — separate proposal.) - **For the adversarial review pass:** manual operator step — after retro draft lands, operator runs `/review-loop` against the commit that landed the report. Pattern-retrospective does NOT automate this invocation; the skill's workflow ends with "Report drafted at ``; commit and run `/review-loop` for adversarial review." Do NOT copy review-loop's agent definitions into this skill — drifts toward Option B merge (user rejected). - **For cross-iteration finding dedup:** use the ≥0.85 threshold from review-loop's drift-guard (same value, embedded in `lib/repeat_detector.py`). - **For provenance:** use the 5-tuple structure from §5 (existing). ## §15 — Process discipline added in 2026-05: - **Before mining:** write 200-word reflexive bias memo at `/research/studies//_methodology.md`. Pre-register 2-3 expected patterns. - **At retro start:** ```bash python ~/.claude/skills/pattern-retrospective/lib/follow_up_check.py \ --project-root ``` Review pending findings; update `follow_up_status` before proceeding. A finding that a later row **supersedes** is treated as closed and is not listed (the registry is append-only, so the superseded row itself keeps whatever status it was written with). The summary line counts what was hidden; add `--include-superseded` to see those rows. Everything else fails open: an unrecognized `follow_up_status` counts as OPEN and is listed with a warning, never dropped, because losing an open finding is worse than showing a stale one. - **During mining** for each new finding: ```bash python ~/.claude/skills/pattern-retrospective/lib/repeat_detector.py \ --new-claim "" \ --scope this-project \ --project-root ``` Optionally rerun with `--scope all` for cross-project repeats. - **At retro end** for each finding: ```bash python ~/.claude/skills/pattern-retrospective/lib/register_finding.py \ --project-root \ --retro-path \ --project \ --category \ --claim "" \ --evidence-supporting 5 --evidence-contradicting 0 \ --proposed-action "" \ --target-date 2026-06-30 ``` `--confidence` is optional and is best omitted, so it is computed from the counts by the §7 formula. If you pass it, it must agree with the counts or registration is refused: a row whose confidence contradicts the evidence stored beside it overstates that evidence to every later reader. When the finding resolves or corrects an earlier one, add `--supersedes `; that link is what closes the older row. **Commit the registry after registering.** If it is git-tracked, the new row is uncommitted until you do, and a `git restore` or branch switch throws it away without a word. The script prints the exact commit command; run it. - **High-stakes retro** (≥5 findings OR conf≥0.70 OR substrate change OR handoff): **Requires** `ANTHROPIC_API_KEY` env var. Use `--dry-run-no-api` to smoke-test without calling the API. ```bash python ~/.claude/skills/pattern-retrospective/lib/dual_llm_coder.py \ --items --coding-prompt "" ``` Block publish at α<0.80; mark exploratory at 0.67-0.79; publish at ≥0.80.