How do I install the LLM Judge Auditor skill?

Run npx skillmds@latest add ityaadiii/llm-judge-auditor in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is LLM Judge Auditor safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with LLM Judge Auditor?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is LLM Judge Auditor free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published LLM Judge Auditor?

ityaadiii (@ityaadiii) published this skill. Their other Agent Skills are listed on their SkillMD profile.

LLM Judge Auditor by ityaadiii — Agent Skill

Name: LLM Judge Auditor
Author: ityaadiii

LLM Judge Auditor

Audit an LLM-as-judge setup for self-preference, position bias, and lack of human agreement before its scores are trusted. Use when a model grades model output, for pairwise preference evaluations, automated scoring, or when someone reports win rates from an AI judge. Refuses to accept judge scores with no measured human agreement.

ityaadiii Updated Sep 17, 2026

File contents

Auditing the judge

An LLM judge is a model with an eval of its own that nobody ran. Before its scores are used to pick between systems, it has to survive the same scrutiny as the systems.

The refusal

No human agreement number, no judge scores. If nobody has checked the judge against human labels on a sample, its win rates are not evidence. Say that and stop.

The four checks

Self-preference. A judge from the same family as one of the candidates favours it. Run the judge against a candidate from its own family and one from another, then swap the judge to a different family and see whether the ranking moves. If it moves, report both and do not pick the flattering one.
Position bias. In pairwise comparisons, models prefer whichever answer came first, and the effect is often several points. Shuffle presentation order per item and measure it. Report the size of the bias as its own number, not as a footnote.
Human agreement. Sample at least 100 judged items and have humans label them. Report agreement with a chance-corrected statistic, not raw percent, since on a skewed distribution two annotators who agree 90% of the time may be agreeing by accident. Below moderate agreement, the judge is not measuring the thing.
Verbosity and format preference. Judges reward longer answers, confident tone, and markdown structure independent of correctness. Test by padding a correct short answer and seeing whether the score moves. If it does, that is a confound in every number the judge has produced.

Procedure

Run all four before reporting anything. Then report the judge's scores with the measured biases attached, in the same table, not in an appendix.

Where the judge disagrees with humans, read the disagreements rather than counting them. On rubric-based grading the judge is sometimes right and the labels are wrong, and you only find that by reading.

Output contract

judge model and family, candidate model families
self-preference: ranking under two judges from different families
position bias, measured, in points
human agreement on n items, chance-corrected
verbosity sensitivity
an explicit statement of whether these scores should be used to make a decision

What this prevents

A win rate of 62% that is a position effect plus a family preference, cited in a decision to switch providers.

---name: llm-judge-auditordescription: Audit an LLM-as-judge setup for self-preference, position bias, and lack of human agreement before its scores are trusted. Use when a model grades model output, for pairwise preference evaluations, automated scoring, or when someone reports win rates from an AI judge. Refuses to accept judge scores with no measured human agreement.---# Auditing the judgeAn LLM judge is a model with an eval of its own that nobody ran. Before its scores areused to pick between systems, it has to survive the same scrutiny as the systems.## The refusal**No human agreement number, no judge scores.** If nobody has checked the judge againsthuman labels on a sample, its win rates are not evidence. Say that and stop.## The four checks1. **Self-preference.** A judge from the same family as one of the candidates favours it. Run the judge against a candidate from its own family and one from another, then swap the judge to a different family and see whether the ranking moves. If it moves, report both and do not pick the flattering one.2. **Position bias.** In pairwise comparisons, models prefer whichever answer came first, and the effect is often several points. Shuffle presentation order per item and measure it. Report the size of the bias as its own number, not as a footnote.3. **Human agreement.** Sample at least 100 judged items and have humans label them. Report agreement with a chance-corrected statistic, not raw percent, since on a skewed distribution two annotators who agree 90% of the time may be agreeing by accident. Below moderate agreement, the judge is not measuring the thing.4. **Verbosity and format preference.** Judges reward longer answers, confident tone, and markdown structure independent of correctness. Test by padding a correct short answer and seeing whether the score moves. If it does, that is a confound in every number the judge has produced.## ProcedureRun all four before reporting anything. Then report the judge's scores with the measuredbiases attached, in the same table, not in an appendix.Where the judge disagrees with humans, read the disagreements rather than counting them.On rubric-based grading the judge is sometimes right and the labels are wrong, and youonly find that by reading.## Output contract- judge model and family, candidate model families- self-preference: ranking under two judges from different families- position bias, measured, in points- human agreement on n items, chance-corrected- verbosity sensitivity- an explicit statement of whether these scores should be used to make a decision## What this preventsA win rate of 62% that is a position effect plus a family preference, cited in adecision to switch providers.

Frequently asked questions