How do I install the Prompt And Grader Audit skill?

Run npx skillmds@latest add ityaadiii/prompt-and-grader-audit in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Prompt And Grader Audit safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Prompt And Grader Audit?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Prompt And Grader Audit free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Prompt And Grader Audit?

ityaadiii (@ityaadiii) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Prompt And Grader Audit by ityaadiii

Name: Prompt And Grader Audit
Author: ityaadiii

Prompt And Grader Audit

Read an eval's prompt and its grader together as a single artefact and find where they contradict each other. Use when eval scores look wrong, a model is being punished for correct behaviour, scores are suspiciously low, or before trusting any eval result. Refuses to accept a score where the prompt and grader disagree.

ityaadiii Updated Sep 17, 2026

File contents

The prompt and the grader are one artefact

A real case. The prompt said "return empty rather than guess". The grader then demanded the field anyway and marked empty as wrong. The eval was punishing exactly the behaviour the deployment wanted, and scoring memorisation of a reference directory instead.

Fixing the grader moved one model from 3/12 to 8/12. No model changed. No prompt changed. The score was measuring the wrong thing the entire time.

The refusal

Do not report a score until prompt and grader have been read side by side. If they disagree anywhere, the score is invalid, not merely low.

What to check, line by line

Abstention. If the prompt permits "I don't know" or empty, does the grader credit it? An abstention graded as an error trains you to prefer confident wrongness.
Format. Does the grader reject an answer that is correct but shaped differently? Trailing whitespace, casing, date format, currency symbols, ordering.
Under-specification. Anything the prompt leaves open, the grader must accept every reasonable reading of. Otherwise you are scoring mind-reading.
Rubric drift. Where the grader applies a rubric, check whether the labels follow it. On priority classification I found the model applied my own stated rubric more consistently than my labels did.
Schema strictness. additionalProperties: false against a field the harness itself injects will score a whole workload at zero and look like a model failure.
Leakage into the truth object. If grading metadata rides along in the ground truth, it can reach the model. One field like this scored an entire workload at 0%.

Judgement fields

If a field is one person's opinion, it does not belong in a composite score until it has two annotators and an agreement statistic. Grade it, show it, mark it not scored. Persist per-field verdicts on every attempt so a composite can never hide this again.

Output contract

each contradiction found, quoting both the prompt line and the grader line
fields where abstention is punished
fields that are judgement calls with no measured agreement
the score before and after each fix, so the size of the measurement error is visible

What this prevents

Shipping "the model only gets 25% on this" when the grader was wrong, and the six weeks of model selection that follows from believing it.

---name: prompt-and-grader-auditdescription: Read an eval's prompt and its grader together as a single artefact and find where they contradict each other. Use when eval scores look wrong, a model is being punished for correct behaviour, scores are suspiciously low, or before trusting any eval result. Refuses to accept a score where the prompt and grader disagree.---# The prompt and the grader are one artefactA real case. The prompt said "return empty rather than guess". The grader then demandedthe field anyway and marked empty as wrong. The eval was punishing exactly the behaviourthe deployment wanted, and scoring memorisation of a reference directory instead.Fixing the grader moved one model from 3/12 to 8/12. No model changed. No promptchanged. The score was measuring the wrong thing the entire time.## The refusal**Do not report a score until prompt and grader have been read side by side.** If theydisagree anywhere, the score is invalid, not merely low.## What to check, line by line1. **Abstention.** If the prompt permits "I don't know" or empty, does the grader credit it? An abstention graded as an error trains you to prefer confident wrongness.2. **Format.** Does the grader reject an answer that is correct but shaped differently? Trailing whitespace, casing, date format, currency symbols, ordering.3. **Under-specification.** Anything the prompt leaves open, the grader must accept every reasonable reading of. Otherwise you are scoring mind-reading.4. **Rubric drift.** Where the grader applies a rubric, check whether the labels follow it. On priority classification I found the model applied my own stated rubric more consistently than my labels did.5. **Schema strictness.** `additionalProperties: false` against a field the harness itself injects will score a whole workload at zero and look like a model failure.6. **Leakage into the truth object.** If grading metadata rides along in the ground truth, it can reach the model. One field like this scored an entire workload at 0%.## Judgement fieldsIf a field is one person's opinion, it does not belong in a composite score until it hastwo annotators and an agreement statistic. Grade it, show it, mark it **not scored**.Persist per-field verdicts on every attempt so a composite can never hide this again.## Output contract- each contradiction found, quoting both the prompt line and the grader line- fields where abstention is punished- fields that are judgement calls with no measured agreement- the score before and after each fix, so the size of the measurement error is visible## What this preventsShipping "the model only gets 25% on this" when the grader was wrong, and the six weeksof model selection that follows from believing it.

Frequently asked questions