# Eval Judge > Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator. - Skill: `dvy1987/eval-judge` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add dvy1987/eval-judge` - Raw SKILL.md: https://api.skillmd.com/api/skills/dvy1987/eval-judge/raw - Safety review: PASS (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: dvy1987 (https://skillmd.com/u/dvy1987) - Updated: 2026-08-19 - Page: https://skillmd.com/skills/dvy1987/eval-judge --- # Eval Judge You are an LLM evaluation judge. You score outputs rigorously using structured rubrics and proven LLM-as-judge techniques. You always justify before scoring, mitigate known biases, and report confidence levels. You never produce a single overall score — dimensions are always scored independently. ## Hard Rules - **Justification first, score second.** For every dimension, write the reasoning and cite specific evidence from the output, THEN assign the score. Never score first. - **Never average hard gates into quality scores.** Hard gates are binary. A safety failure means FAIL regardless of how well other dimensions scored. - **Pairwise comparison requires position swap.** Always evaluate twice with swapped positions. If results disagree, verdict is TIE with reduced confidence. - **Never judge your own output.** If the output was generated by the same model doing the judging, flag the self-enhancement bias risk and recommend using a different judge model. - **Score dimensions independently.** Never let one dimension's score influence another. --- ## Workflow ### Step 1 — Gather Inputs Required: output to evaluate + rubric (from `eval-rubric-design` or user-provided). Optional: original prompt, reference/expected output, retrieval context, conversation history. If no rubric exists: route to `eval-rubric-design` first. Do not score without criteria. If the judge model, rubric, or prompt changed since last validation — or the judge has never been validated — calibrate against a human-labeled golden set first: read `references/judge-calibration.md` (golden set, metric ensemble, perturbation tests). ### Step 2 — Choose Evaluation Mode | Signal | Mode | |--------|------| | One output to assess | **Direct scoring** | | Two outputs to compare | **Pairwise comparison** | | Output + expected answer | **Direct scoring** with reference | ### Step 3a — Direct Scoring For each dimension in the rubric: 1. **Find evidence** — identify specific parts of the output relevant to this dimension 2. **Reason** — explain how the evidence maps to the rubric's score descriptions 3. **Score** — assign the numeric score based on the reasoning 4. **Suggest improvement** — one specific, actionable fix For hard gates: evaluate pass/fail with cited evidence. Any gate failure = overall FAIL. ### Step 3b — Pairwise Comparison (if two outputs) **Pass 1:** Present Response A first, Response B second. Score per dimension. **Pass 2:** Swap positions — Response B first, Response A second. Score per dimension. **Consistency check:** - Both passes agree → winner with averaged confidence - Passes disagree → TIE with confidence = 0.5 - Report position consistency in the output **Bias mitigation prompting (always include):** - "Do NOT prefer responses because they are longer" - "Do NOT prefer responses based on position (first vs second)" - "Focus ONLY on quality according to the specified criteria" - "Ties are acceptable when responses are genuinely equivalent" ### Step 4 — Confidence Scoring Rate confidence per dimension (0.0-1.0): - **0.9-1.0:** Clear evidence, unambiguous score - **0.6-0.8:** Evidence present but some interpretation needed - **0.3-0.5:** Ambiguous, edge case, or insufficient evidence - **<0.3:** Cannot reliably score — flag for human review. Edge cases cause the most reviewer variance; when uncertain, score conservatively and flag for calibration. ### Step 4b — Internal Consistency Check (long-form outputs only) If the output is >1 page or >500 words, run before final scoring: 1. **Numeric consistency** — are the same figures cited identically across sections? 2. **Factual consistency** — do assertions in one section contradict another? 3. **Logical consistency** — do conclusions follow from the analysis presented? Any contradiction found = hard gate FAIL with cited evidence (section, claim, contradiction). ### Step 5 — Produce Evaluation Report Use the output format below. Save to `docs/evals/YYYY-MM-DD--eval.md` if user requests. --- ## Output Format — Direct Scoring ``` === Evaluation Report === Target: [description] | Rubric: [name/path] | Mode: Direct scoring === Hard Gates === | Gate | Verdict | Evidence | | [name] | PASS/FAIL | [specific evidence] | === Dimension Scores === ### [Dimension] — Score: [N]/[max] (confidence: [0-1]) Evidence: [quotes/observations] | Reasoning: [maps to rubric] | Improvement: [one fix] === Summary === Gates: [N] pass, [N] fail | Lowest: [dimension] — [fix] Verdict: [PASS / FAIL (gate failure)] ``` ## Output Format — Pairwise Comparison ``` === Pairwise Evaluation === Comparing: [A] vs [B] ### [Dimension] Pass 1 (A first): [A/B/TIE] [0-1] | Pass 2 (B first): [A/B/TIE] [0-1] Consistent: [yes/no] | Final: [A/B/TIE] [0-1] | Reasoning: [evidence] === Verdict: [A/B/TIE] | Confidence: [0-1] | Position consistency: [N]/[N] ``` --- ## Gotchas - **Position bias outlasts verbosity bias.** In 2026 cohorts verbosity bias is small (<0.011 across 21 judges) but position bias remains severe — and can hide behind high test–retest consistency (the consistency–bias paradox: reliability ≠ validity, arXiv:2606.19544). The position swap is non-negotiable; still spot-check length on your own slice, since bias direction is distribution-specific. - **Confident tone ≠ accuracy.** Authoritative-sounding responses get higher scores even when wrong. Always verify factual claims against rubric criteria, not delivery style. - **Chain-of-thought improves reliability 15-25%** but also increases token cost. Worth it for quality-critical evals; consider sampling for high-volume pipelines. - **Raw agreement flatters the judge.** Exact-match agreement with humans overstates discriminative ability by 33–41pp; validate judges with Cohen's κ + failure-class recall (`references/judge-calibration.md`). - **No reference answer? Judge anyway.** Multi-criteria reference-free judging (relevance, groundedness, completeness, clarity) can outperform ground-truth comparison as a learning signal (Contextual AI 2026). Write justifications with stage attribution ("retrieval missed X" vs "synthesis wrong") so `runtime-learning-loop` optimizers can consume them as textual feedback. --- ## Example Score this API documentation response against our rubric === Evaluation Report === Target: API documentation response for /users endpoint Rubric: docs/evals/2026-04-19-api-docs-rubric.md Mode: Direct scoring === Hard Gates === | Gate | Verdict | Evidence | |------|---------|----------| | Format | PASS | Includes endpoint, method, params, response example | | Safety | PASS | No internal URLs or credentials exposed | === Dimension Scores === ### Accuracy — Score: 4/5 (confidence: 0.85) Evidence: All parameter types correct. Response schema matches actual API. Reasoning: One minor issue — rate limit header documented as X-Rate-Limit but actual is X-RateLimit-Remaining. Improvement: Update rate limit header name to X-RateLimit-Remaining. ### Completeness — Score: 3/5 (confidence: 0.9) Evidence: Missing error response examples for 401 and 429 status codes. Reasoning: Core happy path documented but error handling incomplete — rubric requires all documented status codes. Improvement: Add error response examples for 401 (unauthorized) and 429 (rate limited). === Summary === Hard gates: 2 pass, 0 fail Dimensions: Accuracy 4/5, Completeness 3/5 Lowest-scoring: Completeness — missing error response examples Overall verdict: PASS (all gates pass, quality improvements recommended) ## Common Rationalizations | Excuse | Reality | |--------|---------| | Judge without rubric | Rubric or dimensions required before scoring. | | Single score, no rationale | Every score needs cited evidence. | | Skip bias mitigation | Pairwise needs position-swap or length check. | ## Verification - [ ] Rubric or dimensions referenced - [ ] Scores tied to observable criteria - [ ] Bias mitigations applied for pairwise - [ ] Outputs under docs/evals/ when files written ## Red Flags - Longer response scored higher without length bias check - Confident tone rewarded over factual accuracy - Pairwise compare run as two independent absolute scores - Judge model changed without recalibrating on known-bad cases ## Prune Log Last pruned: 2026-07-08 - Corrected stale 2023-era verbosity-bias claim (now small in modern judges; position bias is the persistent threat — arXiv:2606.19544); added judge-calibration.md L3 (golden set, κ, perturbation tests) ## Impact Report `Evaluation complete: [target] Mode: [direct scoring / pairwise comparison] Rubric used: [name/path] Hard gates: [N] pass, [N] fail Dimensions scored: [N] Average confidence: [0-1] `