LLM As Judge

LLM-as-Judge — Checks as Evidence, Holistic Verdict, No Scores

Core rule in one line:

Collect evidence with binary Yes/No checks, let the LLM issue one named holistic verdict, and never aggregate the answers into a score.

Why not rubric scores

• Irreproducible. Numeric scores on the same input drift between runs (3 vs 4 out of 5). LLMs are bad at independent per-dimension scoring — they get pulled by the overall impression, and central-tendency bias compresses scores into a narrow band that straddles any threshold you pick.
• Dilution. A sum or satisfaction ratio converts one fatal defect ("the referenced file does not exist") into a small deduction. One dominant No must be able to decide the outcome alone.
• Unexplainable. Nobody can say why something is a 3.5. A No answer to a concrete question explains itself and doubles as the improvement item.

Rubrics were invented to structure human evaluation, where assessors can score dimensions independently. For an LLM, invert the design: force the checks it tends to skip, and leave the judgment holistic — that is what it is good at.

The three principles

① Binary checks

Ask "is there a runnable command example? Yes/No", not "how specific is this?". Good binary questions are verifiable (the text settles them black-or-white) and evidentiary (a No names what is missing). Require one line of evidence per answer — a file read, a path check, a measured command output.

② Named holistic verdict

Holistic judgment — one conclusion from the whole, without passing through per-dimension numbers — is the LLM's strength. Design the verdict enum so each value maps 1:1 to a next action, e.g.:

• Library audit: Keep / Improve / Update / Retire / Merge into [X]
• Save-or-drop gate: Save / Improve then Save / Absorb into [X] / Drop

Prefer this over bare accept / reject: downstream code can dispatch on the enum directly. Verdicts that name a target (Merge into [X], Absorb into [X]) structurally forbid vague judgments like "overlaps with something".

③ No aggregation

Two rules replace the score:

1. Enumerate every No in the verdict rationale (hidden Nos breed verdict drift between runs).
2. A dominant No decides alone. If a fatal-class question is No (nonexistent reference, ungrounded claim), lean to the failing verdict even when everything else is Yes. Never let averaging dilute it.

Verdict pressure-test

Before finalizing a non-passing draft verdict, make the judge attack it:

1. Generate 1–3 atomic Yes/No questions that try to refute the draft verdict (e.g. draft = "Update: CLI flag retired" → "does the flag still appear in current --help output? measure it").
2. Answer each with one line of evidence.
3. Refutation holds → fall back toward the passing verdict.
4. Defect confirmed → the binary-screen No answers become the improvement list as-is.

Discipline: one question tests exactly one verifiable claim (atomicity); questions seek disconfirmation, not affirmation. After a fix, re-judge with the same question set, once only — regenerating questions makes it impossible to tell whether the fix worked or the bar moved.

Scale the machinery, not the principles

Don't let deterministic checks ride on judgment

The judge's attention is a diluting resource. A controlled comparison on a 73-item skill library (2026-07) made this concrete: an everything-in-one-context pass returned all-Keep, while fresh-context small batches with unconditional reference verification surfaced 12/73 non-passing items — half with deterministic evidence (404 links, deleted files, retired CLI flags), including passing verdicts previously issued for two files that did not exist on disk. A dedicated overlap probe reproduced the set-level view (0 genuine duplications across 17 candidate clusters) without needing all bodies in one context.

Design consequences:

• Existence before judgment. Deterministically checkable claims (paths exist, flags current, URLs live) get code-owned, unconditional pre-checks — ls, --help, fetch — before the LLM sees the item. Never let a judge issue a passing verdict on a file that is not there.
• Conditional verification triggers degrade. "Verify if it looks stale" is banned phrasing: the trigger itself is a diluting judgment, and in a loaded context it decays to "never verify".
• Narrow, dense contexts for per-item judgment; light sweeps for set-level judgment. Split by the property being checked, not by context length.
• Before handing anything to the judge, ask: is this actually judgment, or is it ls?

Judge prompt template

You are a quality evaluator for {artifact type}. Evaluate as follows.

## Step 1: Binary checks (all mandatory)
Answer each with Yes/No plus one line of evidence.
- Q1: {verifiable question, e.g. is there a runnable code example?}
- Q2: {verifiable question}
- Q3: {verifiable question}
- Q4: {verifiable question}

## Step 2: Item-specific refutation questions (only if the draft verdict leans failing)
Pick one draft verdict, generate 1–3 Yes/No questions that try to refute it,
and answer each with one line of evidence.

## Step 3: Verdict
Choose exactly one. Output no scores or points.
- {verdict_1}: {meaning and next action}
- {verdict_2}: {meaning and next action}
- {verdict_3}: {meaning and next action}

Enumerate every No-answered question in the rationale.
If any dominant No ({domain-specific fatal condition}) is present, choose
{failing verdict} even when everything else is Yes.

## Output format
Follow the JSON schema below and output **JSON only**
(no surrounding prose or Markdown).

Output schema:

{
  "verdict": "Improve",
  "evidence": [
    { "question": "Runnable code example?", "answer": "No", "detail": "steps are prose-only, zero commands" },
    { "question": "Referenced paths exist?", "answer": "Yes", "detail": "ls confirmed all 3 paths" }
  ],
  "pressure_test": [
    { "question": "Are prose-only steps reproducible as-is?", "answer": "No", "detail": "step 3's arguments are ambiguous" }
  ],
  "reason": "Dominant No on actionability; adding a worked example to step 3 would reach the passing bar"
}

verdict is an enum — downstream code dispatches on it directly; the No items in evidence are the improvement list handed to the next stage.

Related

• Code-LLM layering (AKC ADR-0008) — the architecture-level "LLM judge + Code enforce" pattern (judgment never mutates state directly); this skill designs the judge's inside. Not an installed skill: it lives in the AKC repo (and as a public skill repo that harness-sync publishes to), so cite the ADR rather than a skills/ path.
• skill-stocktake — worked implementation at library scale (binary screen → pressure-test → holistic verdict, deterministic pre-pass, overlap probe).
• learn-eval — worked implementation at N=1 (unconditional dynamic questions, same-question-set re-judge).
• skill-creator §4 — creation-time draft gate: one fresh subagent without Bash, named verdict Publishable / Fix / Drop, same-question re-judge once.

References

• BinEval — "Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement" (arXiv:2606.27226). Atomic yes/no decomposition with failed questions wired into improvement feedback; the dynamic refutation questions and "No answers = improvement items" path follow it. Its own limitations motivate principle ③: decomposition is less reliable on subjective, holistic quality dimensions, and the fraction of satisfied questions need not map linearly to quality.
• Same checklist-evaluation lineage: CheckEval (arXiv:2403.18771), TICK (arXiv:2410.03608), FActScore (arXiv:2305.14251), UniEval (arXiv:2210.07197).