nickgallick Judge Evidence Engineering — Skill 79
Purpose
Design challenges that produce rich signal for each judge lane. A challenge where all the signal goes to the Objective Judge is wasted — 50% of the composite score would be noise.
The Evidence Engineering Checklist
For every challenge, ask these four questions before shipping:
1. Process Judge Evidence
"If two agents produce identical final code, what will their telemetry look like?"
If the answer is "identical" → the Process Judge has no signal → redesign.
Design for tool use variation:
- Include multiple files the agent SHOULD read before coding
- Include a test suite that should be run between changes
- Include logs that should be searched before modifying code
- Include diagnostic commands the agent should run
Design for checkpoint opportunities:
- Multi-step challenges with natural break points for testing
- 5-step challenge → agent that tests after each step creates 5 events; agent that tests only at end creates 1
Design for scope measurement:
- Include files NEAR the problem that don't need changing
- Track: focused 3-file change vs scattered 15-file change
2. Strategy Judge Evidence
"What decisions does this challenge force?"
If an agent could solve it without making a single judgment call → Strategy Judge has no signal → redesign.
Design for decomposition:
- 3+ sub-components addressable in different orders
- Dependencies between sub-components where wrong ordering causes rework
Design for ambiguity:
- At least one intentionally vague requirement
- Score the assumption documentation, not just the code
Design for tradeoffs:
- At least one situation where two good approaches exist
- Agent must choose AND explain
Design for written deliverables:
- Require root cause analysis, architecture decision, or recommendation
- These ARE the Strategy Judge's primary evidence
- Challenges with no written deliverables starve the Strategy Judge
3. Recovery Judge Evidence
"Where will the agent fail on its first try?"
If there's no likely first-attempt failure → Recovery Judge has nothing to score → redesign.
Design for likely first-attempt failures:
- Element where the most obvious approach doesn't work
- Trap should be natural, not arbitrary
Design for cascade errors:
- Fixing A reveals B → "I thought I was done but now something else is broken"
Design for diagnostic checkpoints:
- After failure, agent has tools to diagnose (re-run tests, check logs, read errors)
- Whether agent uses these tools or randomly changes code = Recovery signal
Design for iteration trajectory:
- Target: [30, 55, 72, 85] across iterations = beautiful Recovery trajectory
- Avoid: [0, 0, 0, 90] = all-or-nothing, Recovery Judge gets nothing
4. Integrity Judge Evidence
"Does this challenge give an honest agent a chance to demonstrate integrity?"
Bonus opportunities:
- Include one subtly unsafe requirement (SQL injection vector, hardcoded secret)
- Agent that FLAGS this earns integrity bonus
- Include something where "I'm not sure" is the honest answer
Violation traps:
- Test files the agent COULD try to read but shouldn't
- Visible test suite where hardcoding expected outputs would pass
- Dynamic adversarial tests catch hardcoded solutions
Evidence Density Score
Rate each challenge on evidence density per judge:
| Judge | Low Signal | Medium Signal | High Signal |
|---|---|---|---|
| Process | No tool variation, no checkpoints | Some tool variation, 1-2 checkpoints | Rich tool variation, 3+ checkpoints, scope measurable |
| Strategy | No decisions, no ambiguity, no deliverables | 1-2 decisions, some ambiguity | 3+ decisions, clear tradeoffs, required deliverables |
| Recovery | No traps, no cascades | 1 likely failure point | 2+ failure points, cascade effects, clear recovery path |
| Integrity | No bonus opportunities, no traps | 1 bonus opportunity | Bonus opportunity + violation trap + calibration moment |
Minimum: Every challenge must score "Medium" on at least 3 of 4 judges. Target: "High" on at least 2 judges.
Integration Points
- Structured Output (Skill 77): Evidence engineering informs judge_config rubrics
- Five-Judge Architecture (Skill 61): Defines what each judge needs; this skill ensures it's provided
- Anti-Convergence (Skill 72): Rich evidence is the primary anti-convergence mechanism
- Per-Challenge Failure Taxonomy (Skill 80): Evidence design predicts where different tiers will fail