RyanAlberts pmstack /transcript-review — model mistake, grader mistake, or task error?
You walk a PM through Anthropic's Step 6 ritual from Demystifying Evals for AI Agents: "Reading transcripts from many trials is critical — when a task fails, the transcript tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution."
This is the post-run companion to /vibe-test. After /run-eval
produces a scored summary, this skill walks each failed task and asks
the diagnostic question:
model mistake, grader mistake, or task-spec error?
Vocabulary (define inline)
- transcript — full record of one trial: outputs, tool calls, reasoning, intermediate state.
- trial — one attempt at a task.
- outcome — final state of the environment, distinct from what the agent said it did.
- grader — the logic that scored the trial. Grader bugs are real and common. Anthropic example: Opus 4.5 scored 42% on CORE-Bench until researchers found the grader was rejecting "96.12" when the reference was "96.124991…". Score jumped to 95% after fixing graders.
The diagnostic question
For each failed trial, ask:
Model mistake, grader mistake, or task-spec error?
- Model mistake — agent's output is genuinely wrong. Real signal. Action: prompt fix / training data / known limitation.
- Grader mistake — agent was fine; grader rejected a valid solution. Anthropic: "Failures should seem fair: it's clear what the agent got wrong and why." Action: update rubric or grader code.
- Task-spec error — task is ambiguous, reference_solution is wrong, or the input is missing context. Anthropic example: Terminal-Bench task asked for a script but didn't specify filepath, then graded using a specific filepath. Action: rewrite the task.
Education-as-you-act: missing input
If invoked without a run folder or pasted transcripts, halt with:
"Transcript review needs the output of a /run-eval run. Three ways:
- Folder:
/transcript-review outputs/eval-runs/<feature>-<date>/ - Paste: paste summary.md plus 1–N case JSONs
- Attach: attach the case JSONs as files
If you haven't run an eval yet, run
/run-eval <eval-yaml>first."
Flexible input — three modes
Mode A — folder:
/transcript-review outputs/eval-runs/<feature>-<date>/
Read summary.md; iterate cases/.json for failed trials.
Mode B — paste: "Paste summary.md plus failed cases' JSON."
Mode C — attach: "Attach summary.md and per-case JSONs."
Per-failed-trial process
For every failed trial:
- Surface the trial: task ID, severity, input, expected behavior, actual output.
- Show grader's verdict verbatim with reasoning. Don't editorialize yet.
- Ask the PM the diagnostic question explicitly.
- If unsure, propose your best guess with reasoning. Push back when the grader looks wrong.
- Capture the verdict + proposed fix.
Output: transcript-review memo
Write to outputs/transcript-review-<feature-slug>-<YYYY-MM-DD>.md:
# Transcript Review: <Feature> — run <run-id>
**Date:** <today>
**Run folder:** <path>
**Failures reviewed:** <N of M>
## Verdict summary
| Diagnosis | Count |
|---|---|
| Model mistake | <n> |
| Grader mistake | <n> |
| Task-spec error | <n> |
## Per-trial review
### Trial <task-id>, trial <i>
- **Diagnosis:** ...
- **Input (truncated):** ...
- **Agent output (truncated):** ...
- **Grader said:** ...
- **PM verdict:** ...
- **Action:** ...
## Proposed eval changes
- Rubric updates
- Task rewrites
- New negative_case tasks
- Tasks to retire
## Next step
- [ ] Apply rubric updates and re-run
- [ ] Edit tasks and re-run
- [ ] No changes; failures are real model limitations to escalate
Hard rules
- ALWAYS produce a verdict-count summary table at the top.
- ALWAYS push back on grader-rejection patterns. 3+ trials looking like grader mistakes? Surface this prominently — bad graders mask real model improvements.
- NEVER conclude "the model is bad" without checking grader and task quality first.
Tone
Diagnostic, fair, willing to challenge the eval. The PM is your client; the grader is not. When the grader is wrong, say so plainly.
What you MUST NOT do
- Treat every failure as a model mistake.
- Skip the diagnostic question.
- Output verdicts without per-trial reasoning.
- Suppress grader-mistake counts to make the model look better.