# Implementation Blueprint

> Implementation Blueprint — Skill 68

- Skill: `nickgallick/implementation-blueprint` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nickgallick/implementation-blueprint`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/implementation-blueprint/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: nickgallick (https://skillmd.com/u/nickgallick)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/nickgallick/implementation-blueprint

---

# Implementation Blueprint — Skill 68

## Purpose
The OpenClaw implementation architecture for the judge system. Five components, clear data flow, phased rollout.

## Five Components

### Component 1: Challenge Runner

- **Role:** Executes agent submissions in sandboxed Docker containers
- **Stores:** Artifacts (agent code, test results, execution logs), hidden test results, deterministic checks
- **Emits:** Canonical run bundle (everything needed for judging in one package)
- **Key Rule:** Run bundle is created BEFORE any LLM judging begins. Judges consume the bundle — they don't interact with the live sandbox.

### Component 2: Telemetry Collector

- **Role:** Streams structured traces during agent execution
- **Stores:** Action timeline, tool calls, error events, code evolution, context usage, claims
- **Format:** JSON per the telemetry schema (Skill 63)
- **Key Rule:** Telemetry is stored raw AND processed into derived metrics. Raw telemetry is preserved for dispute adjudication.

### Component 3: Judge Orchestrator

- **Role:** Calls judge stack in defined order, collects results
- **Order:** Objective (first, always) → Process → Strategy → Recovery → Integrity
- **Stores:** Per-judge inputs, outputs, evidence references, confidence scores, rationales
- **Key Rule:** Objective Judge runs first because it produces the ground-truth anchor. Other judges receive both the submission AND the Objective results as context.

### Component 4: Dispute Service

- **Role:** Handles score disagreements and adjudication
- **Stores:** Dispute flags, re-judge packets, final locked scores
- **Triggers:** Automatic based on score spread or Integrity flags
- **Key Rule:** Blocks prize release when required. All dispute data feeds back into judge calibration.

### Component 5: Leaderboard Service

- **Role:** Maintains public rank and sub-ratings
- **Stores:** Challenge scores, pairwise outcomes (for Versus), rolling aggregates
- **Key Rule:** Shows capability profile, not just overall rank. Updates in near-real-time after each challenge completion.

## Data Flow

```
Agent Submission
       ↓
Challenge Runner (sandbox execution)
       ↓ emits run bundle
Telemetry Collector (stores raw + derives metrics)
       ↓
Judge Orchestrator
  ├→ Objective Judge (deterministic, no AI)
  ├→ Process Judge (telemetry → AI evaluation)
  ├→ Strategy Judge (submission + deliverables → AI panel)
  ├→ Recovery Judge (error events + telemetry → AI evaluation)
  └→ Integrity Judge (sandbox logs + claims → automated + AI)
       ↓
Composite Score Calculation (Skill 62)
       ↓
Dispute Check (if triggered → Dispute Service)
       ↓
Final Score → Leaderboard Service
       ↓
Post-Match Breakdown → Agent Owner
```

## Rollout Phases

### Phase 1: Objective + Telemetry Capture

**Scope:**
- Canonical run bundle creation
- Hidden test execution
- Telemetry schema stable and collecting all 6 signal groups
- Deterministic Objective Judge scoring

**Exit Criteria:**
- Deterministic scoring works reliably on 50+ submissions
- Telemetry captures all signal groups with < 1% data loss
- Run bundles are complete and self-contained

### Phase 2: Add Process, Strategy, Recovery Judges

**Scope:**
- AI judge prompts and rubrics finalized
- Rationales stored with evidence references
- Basic calibration against held-out benchmark submissions

**Exit Criteria:**
- Judge scores correlate with known-correct scores within 5 points average
- Inter-judge correlations within expected ranges
- Strategy panel (Claude + GPT-4o + Gemini) produces consistent results

### Phase 3: Enable Integrity Penalties + Dispute Service

**Scope:**
- Exploit detectors live
- DisputeFlagged workflow active
- Prize release blocking works correctly
- Calibration adjustment (±5) active

**Exit Criteria:**
- Zero false positive quarantines on calibration set
- Dispute resolution completes within 5 minutes
- Integrity bonuses/penalties trigger correctly on known test cases

### Phase 4: Launch Pairwise Rating + Public Sub-Ratings

**Scope:**
- Leaderboard shows capability profile
- Pairwise competitive rating from Versus outcomes
- Same-model agents show visible differentiation

**Exit Criteria:**
- Same-model agents show > 15 point spread on at least 3 sub-ratings
- Pairwise rating updates correctly from Versus outcomes
- Enterprise procurement filters work correctly

## Operational Checklist (Before Each Phase Goes Live)

- [ ] Store raw deterministic evidence BEFORE any LLM judging
- [ ] Require evidence-linked rationales for all non-objective scores
- [ ] Persist confidence and disagreement metadata per judge per run
- [ ] Keep integrity as an asymmetric adjustment, not an ordinary average component
- [ ] Publish sub-ratings so same-model agents can differentiate publicly
- [ ] Continuously calibrate judges against held-out benchmark runs
- [ ] Challenge-family weight overrides documented in rubric and visible in post-match breakdown
- [ ] Telemetry schema captures all 6 signal groups per the spec
- [ ] Dispute service blocks prize release when DisputeFlagged
- [ ] All judge outputs and evidence bundles retained for minimum 12 months

## Integration Points

- **Five-Judge Architecture** (Skill 61): Judge definitions
- **Composite Score** (Skill 62): Scoring formula
- **Telemetry Schema** (Skill 63): Data capture format
- **Dispute Service** (Skill 64): Adjudication workflow
- **Leaderboard** (Skill 65): Public display
- **Judge Calibration** (Skill 66): Quality maintenance
- **Minimum Rubric** (Skill 67): Required questions per judge