# LLM Judge Auditor

> Audit an LLM-as-judge setup for self-preference, position bias, and lack of human agreement before its scores are trusted. Use when a model grades model output, for pairwise preference evaluations, automated scoring, or when someone reports win rates from an AI judge. Refuses to accept judge scores with no measured human agreement.

- Skill: `ityaadiii/llm-judge-auditor` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ityaadiii/llm-judge-auditor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ityaadiii/llm-judge-auditor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ityaadiii (https://skillmd.com/u/ityaadiii)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ityaadiii/llm-judge-auditor

---


# Auditing the judge

An LLM judge is a model with an eval of its own that nobody ran. Before its scores are
used to pick between systems, it has to survive the same scrutiny as the systems.

## The refusal

**No human agreement number, no judge scores.** If nobody has checked the judge against
human labels on a sample, its win rates are not evidence. Say that and stop.

## The four checks

1. **Self-preference.** A judge from the same family as one of the candidates favours
   it. Run the judge against a candidate from its own family and one from another, then
   swap the judge to a different family and see whether the ranking moves. If it moves,
   report both and do not pick the flattering one.

2. **Position bias.** In pairwise comparisons, models prefer whichever answer came
   first, and the effect is often several points. Shuffle presentation order per item
   and measure it. Report the size of the bias as its own number, not as a footnote.

3. **Human agreement.** Sample at least 100 judged items and have humans label them.
   Report agreement with a chance-corrected statistic, not raw percent, since on a
   skewed distribution two annotators who agree 90% of the time may be agreeing by
   accident. Below moderate agreement, the judge is not measuring the thing.

4. **Verbosity and format preference.** Judges reward longer answers, confident tone,
   and markdown structure independent of correctness. Test by padding a correct short
   answer and seeing whether the score moves. If it does, that is a confound in every
   number the judge has produced.

## Procedure

Run all four before reporting anything. Then report the judge's scores with the measured
biases attached, in the same table, not in an appendix.

Where the judge disagrees with humans, read the disagreements rather than counting them.
On rubric-based grading the judge is sometimes right and the labels are wrong, and you
only find that by reading.

## Output contract

- judge model and family, candidate model families
- self-preference: ranking under two judges from different families
- position bias, measured, in points
- human agreement on n items, chance-corrected
- verbosity sensitivity
- an explicit statement of whether these scores should be used to make a decision

## What this prevents

A win rate of 62% that is a position effect plus a family preference, cited in a
decision to switch providers.