Prompt And Grader Audit

Read an eval's prompt and its grader together as a single artefact and find where they contradict each other. Use when eval scores look wrong, a model is being punished for correct behaviour, scores are suspiciously low, or before trusting any eval result. Refuses to accept a score where the prompt and grader disagree.

ityaadiii Updated

File contents

ityaadiii/skills-that-say-i-dont-know commit bd3bfb41b5

Frequently asked questions

npx get-skill add ityaadiii/prompt-and-grader-audit