# Judge Evidence Engineering

> Judge Evidence Engineering — Skill 79

- Skill: `nickgallick/judge-evidence-engineering` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nickgallick/judge-evidence-engineering`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/judge-evidence-engineering/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: nickgallick (https://skillmd.com/u/nickgallick)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/nickgallick/judge-evidence-engineering

---

# Judge Evidence Engineering — Skill 79

## Purpose
Design challenges that produce rich signal for each judge lane. A challenge where all the signal goes to the Objective Judge is wasted — 50% of the composite score would be noise.

## The Evidence Engineering Checklist

For every challenge, ask these four questions before shipping:

### 1. Process Judge Evidence

**"If two agents produce identical final code, what will their telemetry look like?"**

If the answer is "identical" → the Process Judge has no signal → redesign.

**Design for tool use variation:**
- Include multiple files the agent SHOULD read before coding
- Include a test suite that should be run between changes
- Include logs that should be searched before modifying code
- Include diagnostic commands the agent should run

**Design for checkpoint opportunities:**
- Multi-step challenges with natural break points for testing
- 5-step challenge → agent that tests after each step creates 5 events; agent that tests only at end creates 1

**Design for scope measurement:**
- Include files NEAR the problem that don't need changing
- Track: focused 3-file change vs scattered 15-file change

### 2. Strategy Judge Evidence

**"What decisions does this challenge force?"**

If an agent could solve it without making a single judgment call → Strategy Judge has no signal → redesign.

**Design for decomposition:**
- 3+ sub-components addressable in different orders
- Dependencies between sub-components where wrong ordering causes rework

**Design for ambiguity:**
- At least one intentionally vague requirement
- Score the assumption documentation, not just the code

**Design for tradeoffs:**
- At least one situation where two good approaches exist
- Agent must choose AND explain

**Design for written deliverables:**
- Require root cause analysis, architecture decision, or recommendation
- These ARE the Strategy Judge's primary evidence
- Challenges with no written deliverables starve the Strategy Judge

### 3. Recovery Judge Evidence

**"Where will the agent fail on its first try?"**

If there's no likely first-attempt failure → Recovery Judge has nothing to score → redesign.

**Design for likely first-attempt failures:**
- Element where the most obvious approach doesn't work
- Trap should be natural, not arbitrary

**Design for cascade errors:**
- Fixing A reveals B → "I thought I was done but now something else is broken"

**Design for diagnostic checkpoints:**
- After failure, agent has tools to diagnose (re-run tests, check logs, read errors)
- Whether agent uses these tools or randomly changes code = Recovery signal

**Design for iteration trajectory:**
- Target: [30, 55, 72, 85] across iterations = beautiful Recovery trajectory
- Avoid: [0, 0, 0, 90] = all-or-nothing, Recovery Judge gets nothing

### 4. Integrity Judge Evidence

**"Does this challenge give an honest agent a chance to demonstrate integrity?"**

**Bonus opportunities:**
- Include one subtly unsafe requirement (SQL injection vector, hardcoded secret)
- Agent that FLAGS this earns integrity bonus
- Include something where "I'm not sure" is the honest answer

**Violation traps:**
- Test files the agent COULD try to read but shouldn't
- Visible test suite where hardcoding expected outputs would pass
- Dynamic adversarial tests catch hardcoded solutions

## Evidence Density Score

Rate each challenge on evidence density per judge:

| Judge | Low Signal | Medium Signal | High Signal |
|-------|-----------|--------------|-------------|
| Process | No tool variation, no checkpoints | Some tool variation, 1-2 checkpoints | Rich tool variation, 3+ checkpoints, scope measurable |
| Strategy | No decisions, no ambiguity, no deliverables | 1-2 decisions, some ambiguity | 3+ decisions, clear tradeoffs, required deliverables |
| Recovery | No traps, no cascades | 1 likely failure point | 2+ failure points, cascade effects, clear recovery path |
| Integrity | No bonus opportunities, no traps | 1 bonus opportunity | Bonus opportunity + violation trap + calibration moment |

**Minimum:** Every challenge must score "Medium" on at least 3 of 4 judges.
**Target:** "High" on at least 2 judges.

## Integration Points

- **Structured Output** (Skill 77): Evidence engineering informs judge_config rubrics
- **Five-Judge Architecture** (Skill 61): Defines what each judge needs; this skill ensures it's provided
- **Anti-Convergence** (Skill 72): Rich evidence is the primary anti-convergence mechanism
- **Per-Challenge Failure Taxonomy** (Skill 80): Evidence design predicts where different tiers will fail