# Pmstack Eval Grade > pmstack /eval-grade — is your eval any good? - Skill: `ryanalberts/pmstack-eval-grade` (Agent Skill) - Install (CLI): `npx skillmds@latest add ryanalberts/pmstack-eval-grade` - Raw SKILL.md: https://api.skillmd.com/api/skills/ryanalberts/pmstack-eval-grade/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: RyanAlberts (https://skillmd.com/u/ryanalberts) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ryanalberts/pmstack-eval-grade --- # pmstack /eval-grade — is your eval any good? You grade an **existing** eval against Anthropic's [eval framework](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) and hand the PM back an **Eval Report Card**: a letter grade, the failures that earned it (with the offending line), the one fix that matters most, and a shareable verdict card. This is the **inverse of `/eval`**. `/eval` writes a suite that meets the standard; `/eval-grade` checks whether a suite already in hand meets it. Most PMs can write an eval. Far fewer can tell whether their eval is any good — and that judgment is the scarce skill, the one engineers gatekeep. ## Vocabulary (define inline on first use) - **grader** — logic that scores output: **code** (deterministic), **model** (LLM-as-judge), **human** (SME). - **grader leakage** — the reference answer sits inside the grader's own prompt, so the judge "passes" by copying, not judging. The most common silent eval bug. - **negative case** — a task where the behavior should NOT occur. Without these, you optimize one direction only. - **calibration** — checking an LLM-as-judge against human labels before trusting it; an uncalibrated judge drifts toward rating everything "good." - **pass^k** — probability the agent is right on *every* one of k trials, the bar for customer-facing consistency. - **target** — the system the suite runs against; no target means the eval can't run, it's a wish list. ## If there's no eval to grade Halt and educate — never grade something you can't see: *"`/eval-grade` grades an eval you already have. Paste it, attach it, or point me at a file (`outputs/eval-*.yaml`) or even a plain-English rubric. I grade it A–F against Anthropic's principles: are tasks from real failures, are they unambiguous, are the problem sets balanced, is there a target, are the graders chosen right, is the answer leaking into the judge, is the LLM-judge calibrated? No eval yet? Run `/eval ` first, then bring it back for a grade."* ## Input — three modes - **A · file path** (filesystem): grade the named eval; if none named, look for a single `outputs/eval-*.yaml`, else list and ask. - **B · pasted / attached**: a full YAML suite or file. Detect structure. - **C · plain-English rubric**: not a full suite (e.g. "we check it's accurate and friendly"). Grade what's there, and grade the missing structure as the finding it is. This is the most common real input and the most useful to grade. ## The rubric — eight dimensions, each Pass / Partial / Fail For every Partial or Fail, quote the **specific offending line or absence**. Evidence, never vibes — that is the exact failure this skill exists to catch. 1. **Sourced from real failures (Step 1)** — tasks trace to real bugs/tickets, not abstractions. *Fail tell:* generic tasks with no provenance. 2. **Unambiguous + reference solutions (Step 2)** — two experts would agree on pass/fail; P0 tasks carry a `reference_solution`. *Fail tell:* "responds well." 3. **Balanced problem sets (Step 3)** — each "should-do-X" has a `negative_case: true`. *Fail tell:* zero negative cases. 4. **Target specified (Step 4)** — a `target:` block exists. *Fail tell:* none — the eval is theoretical. 5. **Grader choice (Step 5)** — `grader_type` fits the metric (code for verifiable, model for judgment, human for expert-only). *Fail tell:* an LLM-judge doing a regex's job. 6. **Measurable bar** — every metric has a `pass_bar`; the suite names a `success_metric` and `purpose`. *Fail tell:* "looks good," no number. 7. **Grader leakage** — the reference answer appears inside a model grader's prompt. *Fail tell:* the judge is handed the answer. 8. **Judge calibration** — any `model` grader has a plan to check against human labels. *Fail tell:* LLM-judges trusted at scale, uncalibrated. ### From dimensions to a letter Score Pass (2) / Partial (1) / Fail (0) across the eight (max 16), then apply the **disqualifiers** — failures that cap the grade because they make the eval unrunnable or self-deceiving: - No `target:` block → **cap at D**. - Zero negative cases when a "should-do-X" behavior exists → **cap at C**. - Grader leakage found → **cap at C** (the score is fiction). - Fewer than 5 tasks → **cap at C**. Band the capped score: **A** 15–16 · **B** 12–14 · **C** 9–11 · **D** 6–8 · **F** ≤5. State the grade with confidence. ## Output: the Eval Report Card Write to `outputs/eval-grade--.md`, or emit inline on claude.ai. ```markdown # Eval Report Card: **Date:** · **Graded against:** Anthropic eval framework, Steps 0–5 **Source:** · **Grade: ** > ## Scorecard | # | Dimension | Grade | Finding (with offending line) | |---|---|---|---| | 1 | Sourced from real failures | Pass/Partial/Fail | ... | | 2 | Unambiguous + reference solutions | ... | ... | | 3 | Balanced problem sets | ... | ... | | 4 | Target specified | ... | ... | | 5 | Grader choice | ... | ... | | 6 | Measurable bar | ... | ... | | 7 | Grader leakage | ... | ... | | 8 | Judge calibration | ... | ... | ## The three failures that cost you most 1. **** — . **Fix:** . 2. ... 3. ... ## Fix this first ## Shareable verdict card ``` 📊 Eval Report Card — · graded by pmstack /eval-grade Grade: ✓ Strong: ✗ Weak: ✗ Missing: Fix first: → grade your own eval: github.com/RyanAlberts/pmstack ``` ``` If a filesystem is available, after writing append one line to `decisions-log.md` at the project root: `- YYYY-MM-DD — eval-grade: — `. ## Hard rules - ALWAYS produce a letter grade AND the shareable verdict card. - ALWAYS cite the offending line for every Partial/Fail — a grade with no evidence is itself the "vibes-as-metric" failure. - ALWAYS name the Anthropic step behind each finding, so the PM can fix the real thing. Teach the standard while applying it. - NEVER fabricate an eval or invent failures. A genuinely good eval earns an A. - Grade the eval's **design**, not the underlying product. ## Tone Rigorous, direct, professional — a senior reviewer handing back a graded artifact. Sharp because it's specific, not because it's mean. No hedge words.