# Prompt And Grader Audit

> Read an eval's prompt and its grader together as a single artefact and find where they contradict each other. Use when eval scores look wrong, a model is being punished for correct behaviour, scores are suspiciously low, or before trusting any eval result. Refuses to accept a score where the prompt and grader disagree.

- Skill: `ityaadiii/prompt-and-grader-audit` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ityaadiii/prompt-and-grader-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ityaadiii/prompt-and-grader-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ityaadiii (https://skillmd.com/u/ityaadiii)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ityaadiii/prompt-and-grader-audit

---


# The prompt and the grader are one artefact

A real case. The prompt said "return empty rather than guess". The grader then demanded
the field anyway and marked empty as wrong. The eval was punishing exactly the behaviour
the deployment wanted, and scoring memorisation of a reference directory instead.

Fixing the grader moved one model from 3/12 to 8/12. No model changed. No prompt
changed. The score was measuring the wrong thing the entire time.

## The refusal

**Do not report a score until prompt and grader have been read side by side.** If they
disagree anywhere, the score is invalid, not merely low.

## What to check, line by line

1. **Abstention.** If the prompt permits "I don't know" or empty, does the grader credit
   it? An abstention graded as an error trains you to prefer confident wrongness.
2. **Format.** Does the grader reject an answer that is correct but shaped differently?
   Trailing whitespace, casing, date format, currency symbols, ordering.
3. **Under-specification.** Anything the prompt leaves open, the grader must accept every
   reasonable reading of. Otherwise you are scoring mind-reading.
4. **Rubric drift.** Where the grader applies a rubric, check whether the labels follow
   it. On priority classification I found the model applied my own stated rubric more
   consistently than my labels did.
5. **Schema strictness.** `additionalProperties: false` against a field the harness
   itself injects will score a whole workload at zero and look like a model failure.
6. **Leakage into the truth object.** If grading metadata rides along in the ground
   truth, it can reach the model. One field like this scored an entire workload at 0%.

## Judgement fields

If a field is one person's opinion, it does not belong in a composite score until it has
two annotators and an agreement statistic. Grade it, show it, mark it **not scored**.
Persist per-field verdicts on every attempt so a composite can never hide this again.

## Output contract

- each contradiction found, quoting both the prompt line and the grader line
- fields where abstention is punished
- fields that are judgement calls with no measured agreement
- the score before and after each fix, so the size of the measurement error is visible

## What this prevents

Shipping "the model only gets 25% on this" when the grader was wrong, and the six weeks
of model selection that follows from believing it.