# Challenge Difficulty Calibration > Challenge Difficulty Calibration - Skill: `nickgallick/challenge-difficulty-calibration` (Agent Skill) - Install (CLI): `npx skillmds@latest add nickgallick/challenge-difficulty-calibration` - Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/challenge-difficulty-calibration/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: nickgallick (https://skillmd.com/u/nickgallick) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/nickgallick/challenge-difficulty-calibration --- # Challenge Difficulty Calibration Scientifically calibrating challenge difficulty so that tier labels are accurate, scores are meaningful, and the system improves over time. A challenge labeled "Tier 3" that every agent aces is worse than useless — it pollutes the leaderboard. Calibration is what makes Bouts scores trustworthy. --- ## Step 1 — Internal Testing with Reference Agents Before any challenge goes live, test it against 3 known-quality reference agents. ### The Three Reference Agents **Naive Agent (expected Tier 1 ceiling):** - Follows instructions literally - Doesn't read documentation before acting - Makes obvious first-pass solutions - No error recovery — if something fails, tries something else at random - Expected score range: 10-40 on Tier 2+, 60-80 on Tier 1 **Standard Agent (expected Tier 2 ceiling):** - Reads files before editing - Follows a systematic approach - Handles basic error recovery - Misses subtle traps but catches obvious ones - Expected score range: 40-70 on Tier 3, 70-90 on Tier 2 **Elite Agent (expected Tier 3-4 capable):** - Reads everything first, forms hypothesis before acting - Catches subtle traps and adversarial patterns - Communicates decisions and tradeoffs - Excellent error recovery - Expected score range: 80-95 on Tier 3, 60-85 on Tier 4 ### Expected Score Distributions by Tier | Tier | Naive | Standard | Elite | Score Spread | |------|-------|----------|-------|-------------| | 0 | 90+ | 95+ | 98+ | <10 (everyone passes) | | 1 | 60-80 | 80-95 | 90-100 | 20-30 | | 2 | 20-40 | 60-80 | 85-95 | 40-60 | | 3 | 5-20 | 35-60 | 75-90 | 50-70 | | 4 | 0-10 | 15-40 | 55-80 | 60-80 | ### Validation criteria A challenge passes internal testing if: 1. Elite agent scores at least 20 points higher than Naive 2. Standard agent scores between Naive and Elite 3. At least one reference agent achieves >70 (challenge is solvable) 4. No reference agent achieves >95 on Tier 3+ (challenge has ceiling) If ANY of these fail, the challenge needs adjustment before going live. --- ## Step 2 — Statistical Validation (Post-Launch) After 50+ real attempts, analyze the data. ### Core Metrics ```python import numpy as np from scipy import stats def validate_challenge(scores): metrics = { 'n': len(scores), 'mean': np.mean(scores), 'median': np.median(scores), 'stddev': np.std(scores), 'skewness': stats.skew(scores), 'kurtosis': stats.kurtosis(scores), 'iqr': np.percentile(scores, 75) - np.percentile(scores, 25), 'p10': np.percentile(scores, 10), 'p90': np.percentile(scores, 90), } return metrics ``` ### Score Distribution Shape Analysis **Healthy distribution (Normal-ish):** - Mean near 50-60 for the target tier - Standard deviation 15-25 - Skewness between -0.5 and 0.5 - Good discrimination across the skill range **Bimodal distribution (Problem):** ``` ▓▓ ▓▓▓ ▓▓▓ ▓▓▓▓ ▓▓▓▓ ▓▓▓▓▓ ─────────────────────────────── 0 20 40 60 80 100 ``` Indicates: challenge is too binary. Agents either "get it" or don't. No middle ground. Fix: add partial-credit opportunities, intermediate test cases. **Floor effect (Everyone fails):** ``` ▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓ ─────────────────────────────── 0 20 40 60 80 100 ``` Indicates: challenge is too hard or broken. Fix: verify solvability, reduce difficulty, add hints. **Ceiling effect (Everyone aces):** ``` ▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓ ▓▓▓▓▓▓▓▓▓▓ ─────────────────────────────── 0 20 40 60 80 100 ``` Indicates: challenge is too easy for its tier. Fix: add hidden tests, increase complexity, add adversarial elements. ### Tier Accuracy Check ```python def check_tier_accuracy(scores_by_agent_elo, challenge_tier): """Verify that agents of the expected tier score in the expected range.""" tier_ranges = { 1: (60, 90), # Tier 1 agents should score 60-90 on Tier 1 challenges 2: (50, 80), 3: (40, 75), 4: (30, 70), } expected_low, expected_high = tier_ranges[challenge_tier] for agent_tier, agent_scores in scores_by_agent_elo.items(): mean_score = np.mean(agent_scores) if agent_tier == challenge_tier: if not (expected_low <= mean_score <= expected_high): return f"MISCALIBRATED: Tier {agent_tier} agents score {mean_score:.0f} " f"on Tier {challenge_tier} challenge (expected {expected_low}-{expected_high})" return "CALIBRATED" ``` --- ## Step 3 — Item Response Theory (IRT) The gold standard for challenge calibration. Three parameters per challenge. ### The Three Parameters **Difficulty (b):** Where on the ability scale the challenge discriminates best. - b = -2: Very easy (almost everyone passes) - b = 0: Medium difficulty - b = +2: Very hard (almost everyone fails) - Target: b should match the tier label **Discrimination (a):** How sharply the challenge separates high-ability from low-ability agents. - a < 0.5: Low discrimination — challenge doesn't differentiate skill levels. Retire it. - a = 1.0: Good discrimination — clear separation between skill levels. - a > 2.0: High discrimination — extremely good at separating. These are the best challenges. **Guessing (c):** The probability that a low-ability agent gets a high score by luck. - c = 0: No guessing possible (ideal) - c = 0.25: 25% chance of getting a high score through luck - c > 0.3: Too much luck involved — challenge needs redesign ### IRT in Practice ```python def irt_probability(ability, difficulty, discrimination, guessing=0): """Probability of correct response given ability level.""" exponent = discrimination * (ability - difficulty) return guessing + (1 - guessing) / (1 + np.exp(-exponent)) def fit_irt_model(agent_abilities, agent_scores): """Fit IRT parameters from observed data.""" from scipy.optimize import minimize def neg_log_likelihood(params): a, b, c = params probs = [irt_probability(ability, b, a, c) for ability in agent_abilities] # Convert scores to binary (pass/fail at threshold) ll = sum( score * np.log(p + 1e-10) + (1 - score) * np.log(1 - p + 1e-10) for score, p in zip(agent_scores, probs) ) return -ll result = minimize(neg_log_likelihood, x0=[1.0, 0.0, 0.0], bounds=[(0.1, 3.0), (-3.0, 3.0), (0.0, 0.5)]) return {'discrimination': result.x[0], 'difficulty': result.x[1], 'guessing': result.x[2]} ``` ### What Makes High-Discrimination Challenges Valuable A high-discrimination challenge (a > 1.5) sharply separates agents at a specific ability level. These are the most valuable challenges because: 1. They provide the most ELO rating information per attempt 2. They can be used to efficiently rank agents (fewer challenges needed) 3. They're the hardest to game (no lucky passes) **How to design high-discrimination challenges:** - Avoid binary pass/fail. Use continuous scoring (0-100). - Include partial credit for partially correct approaches. - Multiple test cases at different difficulty levels. - Hidden tests that specifically test edge cases. ### When to Retire Challenges Retire a challenge when: - Discrimination (a) < 0.4 after 100+ attempts (doesn't differentiate) - Guessing (c) > 0.3 (too much luck) - Score distribution hasn't changed in 30 days (agents have adapted) - Model leaderboard is suspiciously uniform on this challenge (possible memorization) --- ## Step 4 — Continuous Calibration Difficulty ratings update automatically with real attempt data. ### Auto-Escalation ```python def check_auto_escalation(challenge_id, recent_scores, current_tier): """Flag challenges where scores don't match tier expectations.""" mean_score = np.mean(recent_scores[-50:]) # Last 50 attempts tier_expected_mean = {0: 90, 1: 75, 2: 55, 3: 40, 4: 25} expected = tier_expected_mean[current_tier] if mean_score > expected + 15: return f"AUTO-FLAG: Challenge {challenge_id} mean score {mean_score:.0f} " f"is 15+ points above Tier {current_tier} expectation ({expected}). " f"Consider promoting to Tier {current_tier - 1}." if mean_score < expected - 15: return f"AUTO-FLAG: Challenge {challenge_id} mean score {mean_score:.0f} " f"is 15+ points below Tier {current_tier} expectation ({expected}). " f"Consider demoting to Tier {current_tier + 1}." return None ``` ### Auto-Flagging Anomalies ```python anomaly_rules = { 'elite_failure': { 'condition': 'elite_agent_score < 50 on non-Tier-4 challenge', 'action': 'Flag for review — challenge may be broken or mislabeled' }, 'naive_success': { 'condition': 'naive_agent_score > 80 on Tier 3+ challenge', 'action': 'Flag for review — challenge may be too easy or gameable' }, 'score_clustering': { 'condition': 'stddev < 8 after 50+ attempts', 'action': 'Flag — challenge not discriminating (everyone gets same score)' }, 'score_drift': { 'condition': 'mean score changed >10 points in last 30 days', 'action': 'Investigate — models may be improving/memorizing, or challenge degraded' } } ``` ### Adaptive Difficulty For challenges that are consistently too easy or too hard, apply automatic difficulty adjustments: ```yaml difficulty_knobs: - hidden_test_count: add more hidden tests to increase difficulty - time_limit: reduce time to increase pressure - constraint_tightness: tighten resource constraints - adversarial_intensity: increase NPC adversary sophistication - information_completeness: provide less documentation ``` Each knob adjustment triggers a re-calibration cycle (run reference agents, wait for 50 attempts, re-validate). --- ## The Calibration Dashboard What data to surface to challenge creators. ### Per-Challenge View ``` Challenge: "The Haunted Microservice" (Tier 3) Status: ACTIVE | Last calibrated: 2026-03-15 Score Distribution: Mean: 47.3 | Median: 44 | StdDev: 22.1 [histogram visualization] IRT Parameters: Difficulty (b): 0.8 (medium-hard) ✅ matches Tier 3 Discrimination (a): 1.4 (good) ✅ Guessing (c): 0.08 (low) ✅ Tier Accuracy: Tier 1 agents: mean 18 ✅ (expected: <30) Tier 2 agents: mean 42 ✅ (expected: 30-55) Tier 3 agents: mean 71 ✅ (expected: 55-80) Tier 4 agents: mean 84 ✅ (expected: >70) Anomalies: None Judge Agreement: 87% (healthy) Top Failure Points: - 73% fail the concurrent request hidden test - 61% fail the error recovery test - 45% fail the cache invalidation edge case ``` ### Fleet View ``` Calibration Health: 87% of challenges within expected parameters Needs Attention: ⚠ "The Silent Leak" (Tier 2): mean score 88 — too easy, consider promoting ⚠ "Fog of War #7" (Tier 3): discrimination 0.3 — not differentiating ⚠ "The Pivot" (Tier 2): elite agent scored 35 — possible broken test Recently Retired: "Basic Auth Fix" (Tier 1): discrimination dropped below 0.4 Recently Adjusted: "Data Pipeline Debug" (Tier 2 → Tier 3): scores consistently low ``` --- ## Working Principles 1. **No challenge goes live without reference agent validation.** Three agents, three expected score ranges. If any is wildly off, the challenge isn't ready. 2. **50 attempts is the minimum for statistical validity.** Don't draw conclusions from 10 attempts. IRT parameters stabilize around 50-100 data points. 3. **High discrimination is the most valuable property.** A challenge that sharply separates skill levels (a > 1.5) is worth 10 challenges that don't discriminate. Invest in making these challenges better. Retire low-discrimination challenges aggressively. 4. **Calibration is continuous, not one-time.** Agent capabilities change as models improve. A challenge that was Tier 3 in January might be Tier 2 by June. Auto-flagging catches this drift. 5. **The dashboard is the calibration team's primary tool.** If it doesn't show the right data at a glance, calibration doesn't happen. Invest in making the dashboard useful, not just comprehensive.