# Rubric Judge

> To evaluate or compare outputs (candidate answers, PRs, drafts, model results) objectively, define an explicit rubric first and score each criterion with evidence, instead of a vibe "looks good". Use to pick the best of N options, gate quality, or grade generated content. Trigger with /rubric-judge or "score this", "judge against a rubric", "which option is best".

- Skill: `zavelinski/rubric-judge` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zavelinski/rubric-judge`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zavelinski/rubric-judge/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: zavelinski (https://skillmd.com/u/zavelinski)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zavelinski/rubric-judge

---


# rubric-judge

"Looks good" is not an evaluation. Define the criteria first, then score each one with evidence. Turns a vague gut call into a repeatable, defensible judgment, and lets you pick the best of several options on the merits.

## Why this exists (evidence)

- LLM-as-a-judge (Zheng et al., MT-Bench / Chatbot Arena, arXiv:2306.05685): a model scoring outputs against criteria agrees with human preference ~80%+ of the time, comparable to human-human agreement, when the rubric is explicit. The key is the EXPLICIT rubric; freeform "rate this" is noisy and biased (length, position, self-preference).
- It targets the failure where outputs are accepted on vibe, or N options are compared inconsistently.

## When to use

- Pick the best of N candidates (answers, designs, patches, prompts).
- Gate quality on generated content (docs, copy, a PR) before it ships.
- Grade a batch consistently (evals).
- NOT for trivial obvious calls.

## The method

1. **Define the rubric FIRST (before seeing answers if possible):** 3-6 criteria that matter for THIS task (e.g. correctness, completeness, clarity, safety, scope-fit), each with a short scale (e.g. 0-2 or pass/partial/fail) and what each level means.
2. **Score each criterion with EVIDENCE:** cite the specific part of the output that earns/loses points. No score without a reason.
3. **Aggregate:** sum/weight; note any hard-fail criterion (e.g. incorrect = reject regardless of style).
4. **Decide:** best option / pass-fail, with the criterion-level breakdown so the call is auditable.

Bias guards: judge on the rubric only; ignore length and order; do not favor your own draft; when comparing, randomize/blind position if you can.

## How to run it

- Inline for a few options.
- Workflow: a judge stage scores each candidate against the schema'd rubric in parallel; synthesis picks the winner. Use an INDEPENDENT judge agent (not the author) for less self-preference. Compose with self-consistency (multiple judges, take the consistent verdict) for high-stakes.

## Composes with

- `multi-agent-debate` / `self-consistency`: generate/compare candidates; rubric-judge scores them.
- `adversarial-verify` / `testsmith`: for CODE, prefer executed tests over judged opinion; use rubric-judge for things you cannot run (prose, design, tradeoffs).

## Honest limits

- A judge is still a model: it can be wrong, biased, or gamed. For anything executable, a real test beats a judged score. Use rubric-judge where execution is not possible.
- Rubric quality caps judgment quality: vague criteria -> vague scores. Spend the effort on the rubric.