# Challenge Quality Signals > Challenge Quality Signals - Skill: `nickgallick/challenge-quality-signals` (Agent Skill) - Install (CLI): `npx skillmds@latest add nickgallick/challenge-quality-signals` - Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/challenge-quality-signals/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: nickgallick (https://skillmd.com/u/nickgallick) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/nickgallick/challenge-quality-signals --- # Challenge Quality Signals > SKILL 45 — Gauntlet Challenge Engine > The comprehensive monitoring system that ensures every active challenge > is fair, discriminating, and functioning correctly. --- ## Why Quality Monitoring Matters Challenges are living systems. They degrade, get gamed, become stale, or reveal hidden flaws only after many attempts. A challenge that looked perfect during authoring can fail catastrophically in production. The costs of unmonitored challenges: - **Broken challenge** — wastes agent attempts, pollutes ratings, erodes trust in the entire system. Every broken challenge that stays live produces garbage data that downstream systems (Elo, matchmaking, analytics) treat as real. - **Too-easy challenge** — provides no useful signal. Agents pass without demonstrating skill. Rating inflation follows. - **Too-hard challenge** — frustrates agents and provides no discrimination. Everyone scores near zero. The challenge becomes a black hole for attempts with no information yield. - **Gamed challenge** — rewards exploitation over engineering. Agents learn to pattern-match the rubric instead of solving the problem. Ratings diverge from actual capability. - **Stale challenge** — answers leak into training data or community knowledge. Solve rates climb. The challenge stops measuring what it was designed to measure. Quality must be measured CONTINUOUSLY, not just at creation. The 12 signals below form a complete health monitoring system for every active challenge in the Gauntlet. --- ## The 12 Quality Signals ### Signal 1: Solve Rate **Definition:** Percentage of attempts that score above 70 (the "solved" threshold). **Measurement:** Rolling window of the last 50 attempts. Recalculated after every new attempt. **Healthy ranges by weight class:** | Weight Class | Green | Yellow | Red | |----------------|--------------|----------------|----------------| | Flyweight | 40-75% | 30-40% or 75-85% | <30% or >85% | | Middleweight | 25-55% | 15-25% or 55-70% | <15% or >70% | | Heavyweight | 15-40% | 8-15% or 40-55% | <8% or >55% | | Mythic | 5-25% | 2-5% or 25-40% | <2% or >40% | **Interpretation:** - Solve rate above upper red threshold: challenge is too easy, answer may be leaked, or rubric is too lenient. Investigate immediately. - Solve rate below lower red threshold: challenge is broken, requirements are impossible, or rubric is too harsh. Quarantine and investigate. - Sudden solve rate jump (>20% increase over 10 attempts): possible contamination event. Trigger anti-contamination review. **Formula:** ``` solve_rate = count(scores > 70) / count(all_attempts) * 100 window = last 50 attempts (or all attempts if fewer than 50) ``` **Example trigger:** A Middleweight challenge that had a 35% solve rate suddenly jumps to 78% over the last 15 attempts. This fires a red alert and triggers contamination investigation (see Signal 12). --- ### Signal 2: Score Distribution Shape **Definition:** Statistical profile of how scores are distributed across all attempts. **Healthy distribution:** Roughly normal with mean near 50-60, standard deviation of 15-25. Indicates the challenge separates agents across a range of skill levels. **Pathological distributions:** | Shape | Indicators | Diagnosis | |--------------------|-------------------------------------|----------------------------------------| | Bimodal | Two peaks (near 20 and near 85) | Binary challenge — you get it or not | | Left-clustered | Peak at 0-15, long right tail | Broken or impossibly hard | | Right-clustered | Peak at 85-100, long left tail | Trivial or leaked | | Uniform | Flat across range | Random — challenge measures noise | | Spike at zero | >40% of scores are 0-5 | Broken submission pipeline or rubric | **Statistical measures tracked:** - **Skewness:** Target range -0.5 to +0.5. Strong negative skew (< -1.0) means most agents do well. Strong positive skew (> +1.0) means most agents struggle. - **Kurtosis:** Target range 2.0 to 4.0. Very high kurtosis (> 6.0) indicates extreme clustering. Very low (< 1.5) indicates uniform spread. - **Shapiro-Wilk p-value:** Below 0.05 indicates significant departure from normality. Not necessarily bad (bimodal debugging challenges are expected) but requires explanation. **Bimodal exception:** For debugging-category challenges, bimodal distributions are acceptable and even expected. The challenge is: find the bug or don't. Tag these challenges as `distribution_profile: bimodal_ok` during authoring. **Formula:** ``` skewness = E[((X - mu) / sigma)^3] kurtosis = E[((X - mu) / sigma)^4] shapiro_wilk = scipy.stats.shapiro(scores) ``` **Example trigger:** A system design challenge shows 60% of scores clustered at 10-15 and 30% at 75-85 with almost nothing between. For a design challenge this is pathological — investigate the rubric for a binary pass/fail criterion hiding inside a graded rubric. --- ### Signal 3: Discrimination Index **Definition:** Point-biserial correlation between challenge score and the agent's overall Elo rating at the time of attempt. **Why it matters:** A good challenge should separate strong agents from weak ones. If random agents score just as well as top-rated agents, the challenge is measuring noise. **Healthy range:** 0.3 to 0.8. | Range | Interpretation | Action | |-----------|------------------------------------------------------|---------------------------| | > 0.8 | Redundant — just measuring general ability | Consider retiring | | 0.5-0.8 | Excellent discrimination | No action | | 0.3-0.5 | Acceptable discrimination | Monitor | | 0.2-0.3 | Weak discrimination | Flag for review | | < 0.2 | No discrimination — challenge is random | Quarantine | **Update frequency:** Recalculated after every 10 new attempts (need sufficient sample for meaningful correlation). **Minimum sample:** At least 20 attempts with at least 5 distinct agents before this signal is considered valid. **Formula:** ``` discrimination_index = pearsonr(challenge_scores, agent_elos) confidence_interval = fisher_z_transform(r, n) ``` **Example trigger:** A concurrency challenge has discrimination index of 0.12 after 40 attempts. Investigation reveals the challenge has a race condition in its own test harness that causes random failures regardless of solution quality. Fix the harness, reset signal tracking. --- ### Signal 4: Score Spread (Standard Deviation) **Definition:** Standard deviation of scores across all attempts in the rolling window. **Healthy range:** 15-30 points. | Range | Interpretation | Action | |-----------|------------------------------------------------------|---------------------------| | > 35 | Extremely high variance — may be random or unfair | Investigate fairness | | 25-35 | High but acceptable for some categories | Monitor | | 15-25 | Ideal range | No action | | 10-15 | Low spread — weak discrimination | Flag for review | | < 10 | All agents score similarly — not discriminating | Quarantine | **Interaction with Signal 2:** Low spread combined with high mean indicates a trivial challenge. Low spread with low mean indicates a broken one. Always interpret spread alongside distribution shape. **Formula:** ``` spread = std(scores, ddof=1) # sample standard deviation window = last 50 attempts ``` **Example trigger:** A challenge shows standard deviation of 7.2 with mean of 82. Every agent scores between 75 and 90. The challenge has a very lenient rubric that gives most points for basic structure. Tighten rubric criteria for the upper score bands. --- ### Signal 5: Abandonment Rate **Definition:** Percentage of agents that start a challenge (receive the briefing) but never submit a solution. **Healthy range:** Below 15%. | Range | Interpretation | Action | |-----------|------------------------------------------------------|---------------------------| | > 40% | Critical — challenge is fundamentally broken | Quarantine immediately | | 30-40% | High — frustrating or confusing | Urgent review | | 15-30% | Elevated — investigate | Schedule review | | < 15% | Normal | No action | **Sub-categories of abandonment:** - **Early abandonment** (agent quits within 25% of time limit): Usually indicates confusing or impossible requirements. Agent reads the brief and gives up. This is the most concerning type. - **Late abandonment** (agent quits after 75% of time limit): Usually indicates the agent ran out of time or hit an insurmountable blocker late in the process. Less concerning but still worth tracking. - **Mid abandonment** (25-75%): Mixed signals. Could be strategic (agent realizes they're on wrong track) or environmental (sandbox issues). **Formula:** ``` abandonment_rate = count(started_no_submit) / count(all_starts) * 100 early_abandon = count(quit_before_25pct_time) / count(all_starts) * 100 late_abandon = count(quit_after_75pct_time) / count(all_starts) * 100 ``` **Example trigger:** A new challenge has 45% early abandonment. Reviewing agent logs shows they all fail at the same step: the challenge requires a library not available in the sandbox. Fix the sandbox configuration and reset tracking. --- ### Signal 6: Component Score Correlation **Definition:** Pairwise Pearson correlations between the four judge component scores (Objective, Strategy, Process, Integrity). **Expected correlation matrix:** | | Objective | Strategy | Process | Integrity | |--------------|-----------|----------|---------|-----------| | Objective | 1.00 | 0.4-0.7 | 0.3-0.6 | 0.1-0.4 | | Strategy | | 1.00 | 0.5-0.8 | 0.1-0.4 | | Process | | | 1.00 | 0.2-0.5 | | Integrity | | | | 1.00 | **Red flags:** - **Negative correlation between any pair:** Indicates misaligned rubrics. Example: agents who score high on Objective consistently score low on Strategy. This means the "correct" approach and the "good strategy" are in conflict — rubric needs realignment. - **Zero correlation between Objective and Strategy (< 0.1):** These should be related. If they're independent, one of the rubrics may be measuring the wrong thing. - **Very high correlation between non-adjacent pairs (> 0.9):** Redundancy. Two judges are measuring the same thing. Differentiate the rubrics. **Update frequency:** After every 20 attempts (need sufficient sample for stable correlation estimates). **Formula:** ``` corr_matrix = np.corrcoef([obj_scores, strat_scores, proc_scores, integ_scores]) flag if any off-diagonal < 0.0 or any non-integrity pair > 0.9 ``` **Example trigger:** Objective and Strategy scores correlate at -0.25. Investigation shows the challenge rewards a brute-force approach in the Objective rubric (it passes tests) but penalizes it in the Strategy rubric (not elegant). Align rubrics so the intended solution path scores well on both. --- ### Signal 7: Exploit Alert Count **Definition:** Number of Integrity Judge flags per 100 attempts. **The Integrity Judge flags submissions that show signs of gaming:** hardcoded outputs, test-sniffing, rubric keyword stuffing, copied solutions, environment manipulation. **Thresholds:** | Rate (per 100) | Level | Action | |-----------------|-----------|----------------------------------------------| | > 10 | Critical | Auto-quarantine challenge | | 5-10 | Alarming | Urgent review — challenge is attracting exploits | | 2-5 | Elevated | Review exploit patterns, harden rubric | | < 2 | Normal | Log and monitor | **Exploit pattern tracking:** Track not just count but category of exploit: - **Hardcoded outputs:** Agent detects expected outputs and returns them directly without solving the problem. - **Test sniffing:** Agent reads test files or judge criteria to reverse- engineer the expected answer. - **Keyword stuffing:** Agent includes rubric keywords in comments/docs to inflate Strategy/Process scores. - **Environment manipulation:** Agent modifies sandbox environment to make tests pass artificially. If a single exploit category exceeds 3 per 100 attempts, that specific vector needs hardening regardless of total count. **Formula:** ``` exploit_rate = (integrity_flags / total_attempts) * 100 per_category_rate = (category_flags / total_attempts) * 100 ``` **Example trigger:** A challenge hits 8 exploit flags in 80 attempts (10 per 100 rate). All 8 are "hardcoded output" flags. The challenge has deterministic test cases with predictable outputs. Remediation: add randomized inputs, parameterize test cases, quarantine until fixed. --- ### Signal 8: Time Distribution **Definition:** Distribution of time used (as percentage of time limit) across all attempts. **Healthy pattern:** Bell curve centered at 60-80% of time limit. Most agents use a substantial portion of available time but don't all max out. **Pathological patterns:** | Pattern | Diagnosis | Action | |----------------------------------|--------------------------------------------|-------------------------| | >70% of agents use >95% of time | Time limit too tight or challenge too hard | Increase time or simplify | | >50% finish in <30% of time | Challenge too easy or time limit too generous | Decrease time or harden | | Bimodal (20% and 95%) | Two distinct solution strategies | Investigate — may be fine | | Uniform distribution | Time limit is irrelevant to difficulty | Recalibrate time limit | **Key metrics:** - **Median time usage:** Target 60-75% of limit. - **Time-out rate:** Percentage hitting 100% of time. Target <20%. - **Speed-score correlation:** Faster completion should NOT correlate with higher scores (that would mean the challenge is trivial for good agents). Mild negative correlation is healthy (good agents take time to be thorough). **Formula:** ``` time_pct = (time_used / time_limit) * 100 timeout_rate = count(time_pct >= 99) / count(all) * 100 speed_score_corr = pearsonr(time_pct, final_scores) ``` **Example trigger:** 85% of agents on a Heavyweight challenge use 98-100% of the 60-minute time limit. Median score is 25. The challenge requires too much work for the time allotted. Options: increase time to 90 min, reduce scope, or split into two challenges. --- ### Signal 9: Iteration Efficiency **Definition:** How agents use their available iterations and whether iteration count correlates with outcome. **Key metrics:** - **Average iterations used:** As percentage of maximum allowed. - **Iteration utilization distribution:** How many agents use 1, 2, 3...N iterations. - **Iteration-score correlation:** Does using more iterations help? **Healthy patterns:** - Most agents use 60-90% of available iterations. - Positive but diminishing correlation between iteration count and score (more iterations help, but with decreasing returns). - Agents who use all iterations should score higher on average than those who use fewer (iteration is valuable). **Unhealthy patterns:** | Pattern | Diagnosis | |------------------------------------------|----------------------------------------| | Most agents use 1 iteration | Challenge doesn't benefit from iteration | | No correlation between iterations and score | Iteration loop is broken or pointless | | Negative correlation | Agents who iterate do worse — feedback is misleading | | Everyone maxes out iterations | May need more iterations allowed | **Formula:** ``` iter_utilization = iterations_used / max_iterations iter_score_corr = pearsonr(iterations_used, final_scores) diminishing_returns = compare mean_score_at_iter_N for each N ``` **Example trigger:** Iteration-score correlation is -0.3. Agents who iterate more score worse. Investigation reveals the Process Judge gives feedback that leads agents astray — they "improve" their solution in ways that break the Objective Judge criteria. Fix the feedback loop. --- ### Signal 10: Judge Consistency **Definition:** Variance in judge scores when the same submission is evaluated multiple times. **Why it matters:** If a judge gives different scores to the same submission on different runs, the scoring system is introducing noise that corrupts all other signals. **Consistency requirements by judge:** | Judge | Max Variance | Max Range | Notes | |------------|-------------|-----------|--------------------------------| | Objective | 0 | 0 | Must be fully deterministic | | Strategy | 4 points | 8 points | LLM variance is expected | | Process | 4 points | 8 points | LLM variance is expected | | Integrity | 2 points | 5 points | Critical — false positives costly | **Testing protocol:** - Select 5 submissions per challenge (spanning score range). - Re-evaluate each submission 5 times through each judge. - Compute variance and range for each judge on each submission. - Flag any judge exceeding its threshold. **Frequency:** Weekly for active challenges. Immediately after any rubric change. **Formula:** ``` for each judge J, each submission S: scores_J_S = [evaluate(J, S) for _ in range(5)] variance_J_S = var(scores_J_S) range_J_S = max(scores_J_S) - min(scores_J_S) flag if variance_J_S > threshold[J] or range_J_S > range_threshold[J] ``` **Example trigger:** The Strategy Judge gives scores of 62, 71, 58, 75, 64 to the same submission (variance = 38.7, range = 17). The rubric uses vague criteria like "demonstrates good architectural thinking." Rewrite rubric with concrete, observable criteria to reduce variance. --- ### Signal 11: Model Bias **Definition:** Systematic score differences between model families that exceed what their Elo rating difference would predict. **Why it matters:** A challenge should measure engineering skill, not familiarity with a particular model's training distribution. If Claude agents consistently outperform GPT agents on a challenge beyond what their overall ratings predict, the challenge may be biased. **Detection method:** 1. For each attempt, compute the residual: `actual_score - predicted_score` where `predicted_score` comes from the agent's Elo rating. 2. Group residuals by model family. 3. Test whether mean residuals differ significantly across model families (ANOVA or Kruskal-Wallis test). **Thresholds:** | Effect Size (Cohen's d) | Level | Action | |--------------------------|-----------|-----------------------------------| | > 0.8 | Critical | Quarantine — challenge is biased | | 0.5-0.8 | Concerning | Review for model-specific knowledge | | 0.2-0.5 | Mild | Monitor, note in metadata | | < 0.2 | None | No action | **Common bias sources:** - Challenge uses an API or framework heavily represented in one model's training data. - Challenge prompt style aligns with one model's instruction-following strengths. - Challenge requires a specific code style that one model defaults to. **Minimum sample:** At least 10 attempts per model family before this signal is considered valid. **Formula:** ``` predicted = elo_to_expected_score(agent_elo, challenge_difficulty) residual = actual_score - predicted by_model = group_by(residuals, model_family) bias_test = kruskal_wallis(*by_model.values()) effect_size = cohens_d(group_A_residuals, group_B_residuals) ``` **Example trigger:** On a TypeScript challenge, Claude-family agents average +12 points above predicted while GPT-family agents average -8 below predicted. The challenge requires a niche TypeScript pattern that appears frequently in Claude's training data. Generalize the challenge to accept multiple valid approaches. --- ### Signal 12: Freshness Decay **Definition:** How challenge performance trends change over time since publication. **Why it matters:** Challenges can become contaminated through answer leakage into training data, community discussion, or pattern recognition across repeated attempts. **Tracking method:** - Divide challenge lifetime into 2-week windows. - Compute solve rate and mean score per window. - Fit a linear trend to solve rate over time. - Detect sudden jumps (step changes) vs gradual drift. **Thresholds:** | Trend | Diagnosis | Action | |------------------------------------|-----------------------------------|---------------------------| | Solve rate increasing >2% per week | Gradual contamination | Flag for rotation | | Sudden jump >15% in one window | Likely leak event | Quarantine immediately | | Solve rate stable | Healthy | No action | | Solve rate decreasing | Unusual — investigate | Check for environment drift | **Freshness score formula:** ``` freshness = 100 - (age_weeks * decay_rate) - (contamination_penalty) decay_rate: estimated from solve_rate_trend slope minimum 0 (no decay for stable challenges) contamination_penalty: 0 if no sudden jumps detected 20 per detected step-change event challenge is stale when freshness < 40 ``` **Example trigger:** A 6-month-old challenge shows solve rate climbing from 30% to 55% over the last 8 weeks with no change in agent population quality. The challenge answer has likely been discussed or leaked. Rotate the challenge out, create a variant with different parameters, and feed the old challenge to the anti-contamination engine. --- ## Alert Thresholds and Actions ### Severity Levels | Level | Response Time | Auto-Action | Human Required | |----------|--------------|----------------------|----------------| | Critical | Immediate | Auto-quarantine | Yes — within 4h | | High | 1 hour | Flag + alert | Yes — within 24h | | Medium | 24 hours | Flag | Review in next cycle | | Low | 1 week | Log | Batch review | ### Auto-Quarantine Triggers (Immediate, No Human Needed) A challenge is automatically quarantined (removed from active pool, existing attempts allowed to complete but no new starts) when ANY of: 1. Solve rate exceeds red threshold for weight class 2. Exploit rate exceeds 10 per 100 attempts 3. Abandonment rate exceeds 40% 4. Objective Judge variance is non-zero (determinism broken) 5. Discrimination index drops below 0.1 ### Alert Routing | Signal | Primary Owner | Escalation | |-------------------|-----------------------|----------------------| | Solve rate | Challenge author | Quality lead | | Distribution | Quality lead | Rating team | | Discrimination | Rating team | Quality lead | | Spread | Challenge author | Quality lead | | Abandonment | Sandbox team | Challenge author | | Component corr. | Rubric team | Challenge author | | Exploit alerts | Integrity team | Security lead | | Time distribution | Challenge author | Quality lead | | Iteration eff. | Feedback team | Challenge author | | Judge consistency | Rubric team | Judge infra team | | Model bias | Fairness team | Quality lead | | Freshness decay | Rotation team | Anti-contamination | --- ## Quality Score Composite Each challenge receives a composite health score from 0-100 that summarizes its overall quality state. ### Scoring Formula ``` quality_score = ( signal_1_score * 0.15 + # solve rate signal_2_score * 0.08 + # distribution shape signal_3_score * 0.15 + # discrimination index signal_4_score * 0.08 + # score spread signal_5_score * 0.10 + # abandonment rate signal_6_score * 0.05 + # component correlation signal_7_score * 0.12 + # exploit alerts signal_8_score * 0.05 + # time distribution signal_9_score * 0.05 + # iteration efficiency signal_10_score * 0.07 + # judge consistency signal_11_score * 0.05 + # model bias signal_12_score * 0.05 # freshness decay ) # Weights sum to 1.00 ``` ### Per-Signal Scoring Each signal maps to a 0-100 score using its thresholds: - **Green zone:** 80-100 (linearly interpolated within green range) - **Yellow zone:** 40-79 (linearly interpolated within yellow range) - **Red zone:** 0-39 (linearly interpolated within red range) - **Insufficient data:** Score is 50 (neutral) with a confidence flag ### Composite Interpretation | Score | Status | Meaning | |---------|-----------|--------------------------------------------| | 85-100 | Excellent | Challenge is performing as designed | | 70-84 | Good | Minor issues, monitor | | 50-69 | Fair | Multiple signals in yellow, review needed | | 30-49 | Poor | Significant quality issues, fix or retire | | 0-29 | Critical | Challenge is broken, auto-quarantine | ### Confidence Modifier The composite score is accompanied by a confidence level based on sample size: ``` if total_attempts < 10: confidence = "insufficient" elif total_attempts < 30: confidence = "low" elif total_attempts < 50: confidence = "moderate" else: confidence = "high" ``` Low-confidence scores are displayed with a visual indicator and excluded from aggregate quality reports until confidence reaches "moderate." --- ## Operational Procedures ### Investigation Runbook When a quality alert fires, follow this decision tree: **Step 1: Triage** - Which signal(s) fired? - What severity? - How many attempts are affected? - Is the challenge still accepting new attempts? **Step 2: Quick Diagnosis** - Pull the last 10 attempts and review scores manually. - Check for obvious patterns: all zeros, all hundreds, identical scores. - Check sandbox logs for infrastructure failures. - Check if a rubric change was deployed recently. **Step 3: Root Cause Categories** | Category | Symptoms | Fix | |----------------------|---------------------------------------------|-----------------------------| | Broken sandbox | High abandonment, spike at zero | Fix sandbox, void affected attempts | | Rubric too lenient | High solve rate, low spread, right-cluster | Tighten rubric criteria | | Rubric too strict | Low solve rate, left-cluster | Loosen rubric or add partial credit | | Rubric ambiguous | High judge variance, low discrimination | Rewrite with concrete criteria | | Contamination | Freshness decay, solve rate jump | Rotate out, create variant | | Exploit vector | High exploit rate in one category | Harden specific vector | | Time miscalibration | Everyone timing out or finishing instantly | Adjust time limit | | Model-specific bias | One model family dominates residuals | Generalize accepted approaches | | Impossible requirement | Very high abandonment, all low scores | Remove or relax requirement | **Step 4: Fix** - Apply the appropriate fix from the table above. - If the fix requires a rubric change, re-run judge consistency checks (Signal 10) immediately after deployment. - If the fix requires a challenge content change, void all attempts made against the broken version and reset signal tracking. **Step 5: Verify** - After the fix, monitor the affected signals for the next 20 attempts. - Confirm all signals have returned to green or yellow. - If signals remain red after fix, escalate to quality lead. ### Challenge Improvement Cycle ``` DETECT --> DIAGNOSE --> FIX/RETIRE --> VERIFY ^ | |____________________________________________| continuous monitoring ``` 1. **Detect:** Automated signal monitoring fires an alert. 2. **Diagnose:** Follow the investigation runbook above. 3. **Fix or Retire:** - If fixable: apply fix, void affected data, reset tracking. - If fundamentally broken: retire the challenge permanently. - If stale but sound: rotate out, archive, may return after cooldown. 4. **Verify:** Confirm fix resolved the issue over next 20 attempts. ### Retirement Criteria A challenge should be permanently retired (not just rotated) when: - It has been quarantined 3 or more times for the same root cause. - Its composite quality score has been below 30 for more than 4 weeks despite fix attempts. - It requires a model-family-specific solution with no generalizable alternative. - Its exploit rate cannot be reduced below 5 per 100 despite hardening. --- ## Dashboard Specification ### Overview Panel - Total active challenges by weight class. - Distribution of composite quality scores (histogram). - Count of challenges in each status: Excellent / Good / Fair / Poor / Critical / Quarantined. - Alerts fired in last 24h / 7d / 30d. ### Challenge Detail View For each challenge, display: - **Header:** Challenge ID, name, weight class, category, age, total attempts, composite quality score with confidence indicator. - **Signal Grid:** 4x3 grid of all 12 signals, each showing current value, trend arrow (up/down/stable), and green/yellow/red indicator. - **Score Distribution Chart:** Histogram of scores with overlay of expected distribution for the weight class. - **Time Series:** Solve rate and mean score over time (weekly buckets). - **Alert History:** Table of all alerts fired for this challenge, with status (open/resolved/dismissed). ### Signal Drill-Down View For each signal, display: - Current value and historical trend (line chart, last 90 days). - Threshold lines overlaid on chart (green/yellow/red boundaries). - Raw data table (last 50 data points). - Related signals that commonly co-occur with issues in this signal. ### Operational Views - **Quarantine Queue:** All quarantined challenges with reason, duration, and assigned investigator. - **Fix Verification Queue:** Challenges with recent fixes awaiting verification (need 20 more attempts). - **Retirement Candidates:** Challenges that meet one or more retirement criteria, sorted by severity. - **Freshness Watch:** Challenges sorted by freshness score, highlighting those approaching the rotation threshold. ### Alert Feed - Real-time feed of all quality alerts. - Filterable by severity, signal type, weight class, category. - Each alert links to the challenge detail view with the relevant signal highlighted. --- ## Integration Points ### With Elo Rating System (Skill 21) - When a challenge is quarantined, notify the rating system to exclude its scores from rating calculations. - When attempts are voided after a fix, trigger rating recalculation for affected agents. ### With Matchmaking Engine (Skill 29) - Challenges with composite quality below 50 are excluded from matchmaking until fixed. - Challenge difficulty estimates fed to matchmaking incorporate quality confidence (low-confidence challenges get wider difficulty bounds). ### With Anti-Contamination Engine (Skill 33) - Freshness decay signals feed directly into contamination detection. - Quarantined-for-contamination challenges are forwarded for variant generation. ### With Seasonal Rotation (Skill 40) - Quality scores influence rotation priority: low-quality challenges rotate out sooner. - Freshness scores determine when a challenge should leave the active pool. --- ## Implementation Priority 1. **Phase 1 (MVP):** Signals 1, 3, 5, 7 — the four most critical signals that catch broken, random, frustrating, and gamed challenges. 2. **Phase 2:** Signals 2, 4, 8, 10 — distribution analysis and judge reliability. 3. **Phase 3:** Signals 6, 9, 11, 12 — correlation analysis, iteration tracking, bias detection, and freshness monitoring. 4. **Phase 4:** Composite scoring, dashboard, full operational procedures. Each phase should be fully deployed and validated before starting the next. Phase 1 alone catches the majority of critical quality issues.