rubric-judge
"Looks good" is not an evaluation. Define the criteria first, then score each one with evidence. Turns a vague gut call into a repeatable, defensible judgment, and lets you pick the best of several options on the merits.
Why this exists (evidence)
- LLM-as-a-judge (Zheng et al., MT-Bench / Chatbot Arena, arXiv:2306.05685): a model scoring outputs against criteria agrees with human preference ~80%+ of the time, comparable to human-human agreement, when the rubric is explicit. The key is the EXPLICIT rubric; freeform "rate this" is noisy and biased (length, position, self-preference).
- It targets the failure where outputs are accepted on vibe, or N options are compared inconsistently.
When to use
- Pick the best of N candidates (answers, designs, patches, prompts).
- Gate quality on generated content (docs, copy, a PR) before it ships.
- Grade a batch consistently (evals).
- NOT for trivial obvious calls.
The method
- Define the rubric FIRST (before seeing answers if possible): 3-6 criteria that matter for THIS task (e.g. correctness, completeness, clarity, safety, scope-fit), each with a short scale (e.g. 0-2 or pass/partial/fail) and what each level means.
- Score each criterion with EVIDENCE: cite the specific part of the output that earns/loses points. No score without a reason.
- Aggregate: sum/weight; note any hard-fail criterion (e.g. incorrect = reject regardless of style).
- Decide: best option / pass-fail, with the criterion-level breakdown so the call is auditable.
Bias guards: judge on the rubric only; ignore length and order; do not favor your own draft; when comparing, randomize/blind position if you can.
How to run it
- Inline for a few options.
- Workflow: a judge stage scores each candidate against the schema'd rubric in parallel; synthesis picks the winner. Use an INDEPENDENT judge agent (not the author) for less self-preference. Compose with self-consistency (multiple judges, take the consistent verdict) for high-stakes.
Composes with
multi-agent-debate / self-consistency: generate/compare candidates; rubric-judge scores them.adversarial-verify / testsmith: for CODE, prefer executed tests over judged opinion; use rubric-judge for things you cannot run (prose, design, tradeoffs).
Honest limits
- A judge is still a model: it can be wrong, biased, or gamed. For anything executable, a real test beats a judged score. Use rubric-judge where execution is not possible.
- Rubric quality caps judgment quality: vague criteria -> vague scores. Spend the effort on the rubric.
1---2name: rubric-judge3description: To evaluate or compare outputs (candidate answers, PRs, drafts, model results) objectively, define an explicit rubric first and score each criterion with evidence, instead of a vibe "looks good". Use to pick the best of N options, gate quality, or grade generated content. Trigger with /rubric-judge or "score this", "judge against a rubric", "which option is best".4---56# rubric-judge78"Looks good" is not an evaluation. Define the criteria first, then score each one with evidence. Turns a vague gut call into a repeatable, defensible judgment, and lets you pick the best of several options on the merits.910## Why this exists (evidence)1112- LLM-as-a-judge (Zheng et al., MT-Bench / Chatbot Arena, arXiv:2306.05685): a model scoring outputs against criteria agrees with human preference ~80%+ of the time, comparable to human-human agreement, when the rubric is explicit. The key is the EXPLICIT rubric; freeform "rate this" is noisy and biased (length, position, self-preference).13- It targets the failure where outputs are accepted on vibe, or N options are compared inconsistently.1415## When to use1617- Pick the best of N candidates (answers, designs, patches, prompts).18- Gate quality on generated content (docs, copy, a PR) before it ships.19- Grade a batch consistently (evals).20- NOT for trivial obvious calls.2122## The method23241. **Define the rubric FIRST (before seeing answers if possible):** 3-6 criteria that matter for THIS task (e.g. correctness, completeness, clarity, safety, scope-fit), each with a short scale (e.g. 0-2 or pass/partial/fail) and what each level means.252. **Score each criterion with EVIDENCE:** cite the specific part of the output that earns/loses points. No score without a reason.263. **Aggregate:** sum/weight; note any hard-fail criterion (e.g. incorrect = reject regardless of style).274. **Decide:** best option / pass-fail, with the criterion-level breakdown so the call is auditable.2829Bias guards: judge on the rubric only; ignore length and order; do not favor your own draft; when comparing, randomize/blind position if you can.3031## How to run it3233- Inline for a few options.34- Workflow: a judge stage scores each candidate against the schema'd rubric in parallel; synthesis picks the winner. Use an INDEPENDENT judge agent (not the author) for less self-preference. Compose with self-consistency (multiple judges, take the consistent verdict) for high-stakes.3536## Composes with3738- `multi-agent-debate` / `self-consistency`: generate/compare candidates; rubric-judge scores them.39- `adversarial-verify` / `testsmith`: for CODE, prefer executed tests over judged opinion; use rubric-judge for things you cannot run (prose, design, tradeoffs).4041## Honest limits4243- A judge is still a model: it can be wrong, biased, or gamed. For anything executable, a real test beats a judged score. Use rubric-judge where execution is not possible.44- Rubric quality caps judgment quality: vague criteria -> vague scores. Spend the effort on the rubric.
zavelinski