Cc Canary

cc-canary — long-form regression writeup

Primary question: has Claude regressed on this user's work, and when?

Bundled script scripts/compute_stats.py does ~95% of the work in ~2.5s: scans JSONLs, runs inflection + transition-day detection, builds pre/post aggregates, cross-version comparison, hour-of-day, word frequency, three- period thinking depth, visibility transition, per-turn rates, and abnormalities — then renders a complete markdown skeleton with every table filled. Narrative slots are marked <!-- C: ... -->.

Default window: 60d. Accept 7d / 14d / 30d / 60d / 90d / 180d.

Framing — three-bucket classification

1. Model-side — same user/task, worse outcomes; cross-version worse; reasoning-depth dropping
2. User-side — project-mix shift; shorter/imperative prompts; new codebase; shortcut vocab rising
3. Ambiguous — mixed confounds, borderline effect size, either-way-explainable

Your 3-step job

1. Run the script

Bash(python3 <SKILL_DIR>/scripts/compute_stats.py --window {window} --render-md /tmp/cc-canary-skeleton-{window}.md > /dev/null 2>&1)

<SKILL_DIR>: .claude/skills/cc-canary/ → fallback to ~/.claude/skills/cc-canary/.

Flags: --window {Nd} (required); --include-agents (include subagent sessions); --min-user-words N (default 10).

If the script fails: report error, retry once with --include-agents, else stop. Never fall back to hand-computation — that's the slow path.

2. Read the skeleton

Read /tmp/cc-canary-skeleton-{window}.md

3. Fill every <!-- C: ... --> placeholder and save

Write ./cc-canary-{YYYY-MM-DD}.md

End your message with the absolute path: Wrote /Users/.../cc-canary-{date}.md · paste-ready.

Narrative placeholders

Each placeholder's inline comment already spells out what to write. Summary:

• verdict-line — HOLDING / SUSPECTED REGRESSION / CONFIRMED REGRESSION / INCONCLUSIVE + brief justification
• summary — 2–4 sentences: verdict + inflection + biggest pre→post delta. No counter-evidence hedging.
• timeline — 1–2 paragraphs on the daily series shape
• xv-para — 1 paragraph on cross-version (if §3 present)
• finding-N-class × up to 5 — inline classification label
• finding-N-reason × up to 5 — 2–3 sentences max, evidence-first. Lead with the strongest number (cross-version Δ, §2 value, appendix rate). No signal-line restating, no rhetorical buildup
• root-cause — 3–5 paragraphs tying strongest signals together
• what-would-help — 2–4 concrete bullets
• appendix-a1…a4, b, c, d, e, f, g, h — 1 paragraph each (see table context in skeleton)
• meta-note — 2–5 sentences, first person, honest, no claimed feelings, acknowledge the recursion

Verdict calibration

• HOLDING: ≤1 model-side signal
• SUSPECTED REGRESSION: 2–3 model-side signals
• CONFIRMED REGRESSION: ≥3 model-side signals + non-empty cross-version showing decline + session_count ≥ 15 + ≥2 models + inflection.gap_sigma ≥ 1.0
• INCONCLUSIVE: session_count < 15 OR inflection.method == "fallback_split_half" with overlapping confounds

Cap at SUSPECTED when: only one model; <15 sessions; single-project with project starting mid-window; inflection coincides with a visible user-side event.

All the data you need (session_count, model mix, inflection method, cross-version presence) is rendered as plain text in the skeleton.

Grounding example (for finding-N-reason)

Classification: model-side. Read:Edit dropped 9.0 → 1.0 (-89%, concerning) while cross-version shows opus-4-7 at 0.39 vs opus-4-6 at 1.00 on the same user's workload. No project-mix shift near the inflection — model is defaulting to edit-first.

Hard rules

• Never read, grep, or glob ~/.claude/projects/**/*.jsonl. Never run jq/awk/wc on session files. Script owns all that.
• Never touch tables or numbers — they came from real data.
• Every finding gets a classification label.
• Hedge when cross-version is empty or session_count < 15.
• Do not verdict CONFIRMED REGRESSION without the full checklist.
• Do not save the skeleton as-is — replace every <!-- C: ... --> first.

Failure modes

• Script import error → check python3 -V ≥ 3.8; retry once with --include-agents; else stop.
• Skeleton < 5KB → likely no sessions in window. Check script error.
• inflection.method == fallback_split_half → state it; cap at SUSPECTED.
• Cross-version Δ None → div-by-zero when model-A value is 0; note the confound.