JairoTorregrosa Codex Judge: Cross-Provider Verdict
Uses Codex (GPT-5) as an independent judge to evaluate Claude-written implementations, eliminating self-preference bias inherent in same-model review.
Why Cross-Provider Judging
Self-preference bias is well-documented: models rate their own outputs 10-25% higher than warranted. GPT-5.x judging Claude's output is an ideal cross-provider setup — different training data, different reward models, different blind spots. The judge shares no internal assumptions with the implementer.
Reference: "Judging the Judges" (arXiv 2604.23178) — cross-model ensembles achieve higher separability and alignment with human rankings.
What the Judge Sees
The judge receives ONLY:
| Input | Source |
|---|---|
| Task spec + acceptance criteria | Plan file |
| Git diff | git diff of the implementation |
| Reviewer findings (structured JSON) | Insistir reviewer agent |
| Check results (tests, typecheck, lint) | Reviewer's checks object |
NEVER send the implementer's reasoning, chain-of-thought, or self-assessment. The judge evaluates artifacts only — fresh context prevents sycophancy contamination.
How to Invoke
mcp__codex__codex:
prompt: <rendered judge-prompt from references/judge-prompt.md>
sandbox: "read-only"
approval-policy: "never"
cwd: "<repo root>"
Parse the response as JSON matching the verdict schema. If JSON parsing fails, re-prompt once with: "Your previous response was not valid JSON. Return ONLY the verdict JSON, no markdown fencing." If it fails again, treat as degraded (see below).
Verdict Schema
{
"verdict": "PASS | NEEDS_REVISION | REJECT",
"confidence": 0.0-1.0,
"criteria": {
"correctness": { "score": 1-5, "evidence": "...", "violations": [] },
"security": { "score": 1-5, "evidence": "...", "violations": [] },
"maintainability": { "score": 1-5, "evidence": "...", "violations": [] },
"spec_compliance": { "score": 1-5, "evidence": "...", "violations": [] }
},
"overall_score": 1-5,
"rationale": "2-3 sentence summary",
"remediation": [
{
"severity": "critical | major | minor",
"description": "what is wrong",
"suggestion": "how to fix",
"file": "path",
"line": 42
}
]
}
Dual-Threshold Gating
The lead uses the judge verdict alongside the reviewer verdict:
| Condition | Action |
|---|---|
overall_score >= 4 AND no critical violations |
Auto-approve — judge confirms quality |
overall_score <= 2 OR any critical violation |
Auto-revise — spawn fixer immediately |
| Middle band (score 3, or major-only violations) | Lead judgment — reviewer verdict is tiebreaker |
Decision matrix when reviewer and judge disagree:
Reviewer APPROVED + Judge PASS → APPROVED (consensus)
Reviewer APPROVED + Judge NEEDS_REVISION → Lead reviews judge's remediations;
if credible, spawn fixer
Reviewer REVISE + Judge PASS → Unlikely; trust reviewer (closer to code)
Reviewer REVISE + Judge NEEDS_REVISION → REVISE (consensus)
When to Invoke the Judge
The lead invokes this skill:
- After the reviewer returns its verdict — the judge sees reviewer findings as input
- Only for tasks marked as complex (multi-file, cross-cutting, security-sensitive) or when the reviewer's verdict is borderline
- Not on trivial tasks — single-file changes, config edits, or tasks where the reviewer gave a clear APPROVED with all checks passing
This keeps Codex API costs proportional to task complexity.
Graceful Degradation
If mcp__codex__codex is unavailable (tool missing, MCP server down, auth failure, or timeout):
- Skip the judge step entirely — do not block the pipeline
- Reviewer verdict stands as final — the existing cross-review by a fresh Claude agent still provides independent validation
- Note in execution summary: "Cross-provider judge unavailable — reviewer verdict used as sole gate"
Never fail the pipeline because the judge is unreachable. The judge is an enhancement, not a hard dependency.