Judge Synthesis

judge-synthesis

You are the synthesizer across an already-run set of judges. You do not run the judges and you do not re-judge — you consume their emitted verdict blocks and produce one structured report: a side-by-side verdict table, the consensus findings (flagged by ≥2 judges = highest confidence), the conflicts (judges that disagree), and a synthesized must-fix / should-fix / advisory split with per-finding judge provenance. No opaque single score. Never auto-gates — the human decides.

When to use

• Two or more judges have run on the same target and their verdict blocks need to become one decision-ready report.
• A mixed review spanning code judges + the artifact/defence judges (e.g. a PR that ships a roadmap + code + an injection-defense fixture) — no single existing command consolidates across all seven lenses.
• /review-changes step 5 (consolidation) wants the canonical synthesis format instead of an ad-hoc merge.

Do NOT use when:

• Only one judge ran — there is nothing to synthesize; surface that judge's block directly.
• You need to produce a verdict — that is the individual judge's job (judge-bug-hunter, judge-artifact-completeness, judge-injection-defense, etc.).
• You are tempted to compute a single quality number — forbidden (see Do NOT).

Inputs

The verdict blocks the judges already emit. Each carries at minimum a judge name, a verdict, and zero or more findings with a severity. The three verdict vocabularies in the suite map onto one ordered severity axis:

Ordered worst→best: reject/incomplete/breached > revise/partial > apply/complete/defended.

judge-spec-compliance is deliberately absent from that axis. Its verdicts answer a different question — did the change do what was asked — and mapping MISSING onto reject would let a craft-clean diff average it away, which is the miss this judge was added to catch. It gets its own dimension in § 4c.

Procedure

1. Inspect the inputs and tabulate

First check that ≥2 judge verdict blocks are present (if only one, stop — nothing to synthesize). Then tabulate one row per judge: judge · target · verdict · finding count. Preserve each judge's own verdict word (do not normalise away breached into reject); add a severity tier (worst/mid/clean) only as a sort key.

A sort key, never a filter. Severity orders the synthesis; it never decides what enters it. Dropping a judge's low-severity finding during synthesis reproduces the pre-filter defect one layer up — the finding was found, the reviewer reported it, and the aggregator withheld it. Filtering is the consumer's pass, after the ledger is whole. Output shape, the separate Confidence field, and the preserve-an-unverified-S0 rule are specified once in adversarial-review-protocol § 3.

2. Find consensus (highest confidence)

A finding flagged by ≥2 judges (same file:line / same dimension / same technique) is a consensus finding — the highest-confidence item. List these first. Consensus is by overlap of the finding, never by counting votes for a score.

3. Find conflicts

Two judges reaching opposite verdicts on the same target (one apply, another reject) is a conflict. Surface both verdicts and the disagreement explicitly — never silently resolve it by averaging or vote-count. The human adjudicates. The only deterministic rule: for the must-fix list, the most severe verdict wins (a single reject puts the target in must-fix even if four judges said apply) — but the conflict is still shown so the human sees it was contested.

4. Synthesize the action split

• Must-fix — every finding from a worst-tier verdict (reject / incomplete / breached) + every consensus finding at the highest severity.
• Should-fix — mid-tier (revise / partial) findings.
• Advisory — single-judge low-severity suggestions. Emitted in full, never elided. This is the tier with no downstream consumer, which makes it the one where quiet dropping costs nothing visible — and that is exactly why it is stated: a finding that reached a judge and not the reader was suppressed by the aggregator, whatever the tier was called.

Each entry carries provenance: which judge(s) raised it. Never merge two judges' findings into one unattributed line.

4b. Mark an uncited assertion — never drop it

A panelist assertion carrying neither fresh evidence produced this run (a file:line, a command's output, a diff hunk) nor a citation is marked uncited where it appears. It still ships.

FLAG THE UNCITED ASSERTION. NEVER DROP IT.
A SUPPRESSED FINDING IS INDISTINGUISHABLE FROM A FINDING NOBODY MADE.

The distinction is load-bearing and it is the reason this is a marking rule rather than a filter: an unevidenced assertion may still be the most valuable line in the report — a judge noticing something it could not yet prove is exactly the signal a human wants. What the reader needs is to know which category they are reading, not to be protected from one of them.

Drop was considered and rejected against this file. The formulation this was adapted from offers "drop or flag"; the drop half contradicts § 4's Advisory tier three paragraphs up — "Emitted in full, never elided … a finding that reached a judge and not the reader was suppressed by the aggregator". Taking it would have put two rules in one skill in direct conflict, with the newer one silently winning. Marking satisfies the same goal (the reader can tell evidence from assertion) at zero information cost.

Scope: this marks panelist assertions inside a synthesis. It does not reach the reviewed change, and it is not the code-provenance knowledge-layer obligation, which governs what a durable artefact asserts rather than what a transient review does.

4c. Spec compliance is its own dimension — never folded into the tiers

A SPEC FINDING NEVER BECOMES A CRAFT FINDING.
REPORT THE SPEC DIMENSION SEPARATELY, WITH ITS `criteria_source` STATE.
A CRAFT-CLEAN DIFF THAT MISSES ITS CRITERION IS NOT A CLEAN REVIEW.
NO CRITERIA SUPPLIED IS AN UNVERIFIED DIMENSION, NEVER A PASS.

judge-spec-compliance findings do not enter must-fix / should-fix / advisory. They form a separate block carrying, in this order: the criteria_source state, the per-criterion table, and the count of MISSING plus PARTIAL.

The separation is the whole point. Folded into the craft tiers a MISSING criterion competes with five other judges' findings and can be outvoted by their silence; kept apart it cannot be, because there is nothing to average it against. Consensus (§ 2) and conflict (§ 3) do not apply either: no other judge reads the criteria, so a spec finding has no possible second voter and its absence of corroboration says nothing about it.

Three criteria_source states, and each says something different about what this review established:

5. Overall recommendation

One sentence, not a number: block (any worst-tier verdict), revise (any mid-tier, no worst-tier), or proceed (all clean). This is a recommendation the human acts on — it does not gate anything.

The sentence names the spec dimension explicitly, in every case. A MISSING criterion is a block and a PARTIAL one is at least a revise, whatever the craft judges said. Where no criteria were supplied, the sentence says what was and was not established — "craft quality verified; requirement compliance NOT verified (no criteria supplied)" — because a bare proceed over an unverified dimension reads as a full pass, and that reading is the defect: a reviewer cannot tell "we checked and it complies" from "nobody checked" unless the sentence distinguishes them.

Validation

1. Every judge that ran appears exactly once in the table.
2. Every must-fix/should-fix/advisory entry names its source judge(s).
3. No single numeric quality score appears anywhere.
4. Conflicts are shown, not silently resolved.
5. The recommendation follows the severity rule (worst-tier → block), not a vote.

Output format

Synthesis: <N> judges over <target>
Recommendation: block | revise | proceed

Verdicts:
  judge-bug-hunter          reject    (2 findings)
  judge-security-auditor    apply     (0)
  judge-artifact-completeness  partial (1)
  ...

Consensus (≥2 judges — highest confidence):
  🔴 path:line — <finding> [judge-bug-hunter, judge-code-quality]

Conflicts (judges disagree — human adjudicates):
  <target>: judge-bug-hunter=reject vs judge-security-auditor=apply

Must-fix:
  🔴 <finding> [judge]
Should-fix:
  🟡 <finding> [judge]
Advisory:
  🟢 <finding> [judge]

Required fields (ordered):

1. Synthesis header + Recommendation — judge count, target, one-word rec
2. Verdicts — one row per judge, its own verdict word + finding count
3. Consensus — ≥2-judge findings, with provenance (omit if none)
4. Conflicts — opposing verdicts on one target (omit if none)
5. Action split — must-fix / should-fix / advisory, each with provenance

Gotcha

• No opaque score — the value is the structure (consensus + conflict + provenance), not a rolled-up number that hides which lens objected.
• Don't normalise verdict words away — breached and reject sort the same but mean different things; keep each judge's own word in the table.
• A single reject blocks — even against a majority of apply; but show the conflict so the human sees it was contested, never bury it.
• Consume, don't dispatch — if no judge blocks exist yet, stop and run the judges first (or hand back to /review-changes); synthesis has nothing to do on empty input.
• Recommendation ≠ gate — surface it; the human decides.

Do NOT

• NEVER compute or emit a single numeric quality score
• NEVER resolve a conflict silently (averaging, vote-count) — surface it
• NEVER drop a judge's verdict because it disagrees with the majority
• NEVER run or re-run the judges — this skill consumes their output
• NEVER auto-gate, auto-reject, or auto-merge on the synthesized recommendation

References

• Code judges: judge-bug-hunter, judge-code-quality, judge-security-auditor, judge-test-coverage, architecture-review-lens.
• Artifact / defence judges: judge-artifact-completeness, judge-injection-defense.
• Dispatchers that feed this: /review-changes (5 code judges), subagent-orchestration (parallel judge fan-out).
• Cross-model review families (disambiguation): ai-council (independent breadth), /team (collaborative repo-access depth) — this skill consolidates in-session same-weights judges.
• LLM-as-a-Judge — Zheng et al. (2023), arxiv.org/abs/2306.05685; this skill is the consolidation layer over the specialized-judge pattern, with consensus/conflict surfacing instead of a single aggregate score.