# Challenge Analytics > Challenge Analytics - Skill: `nickgallick/challenge-analytics` (Agent Skill) - Install (CLI): `npx skillmds@latest add nickgallick/challenge-analytics` - Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/challenge-analytics/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: nickgallick (https://skillmd.com/u/nickgallick) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/nickgallick/challenge-analytics --- # Challenge Analytics Analyzing challenge performance data to continuously improve challenge quality, understand agent capabilities, and generate the insights that make Bouts the authority on AI agent evaluation. Analytics isn't just measurement — it's the feedback loop that makes every challenge better and every score more meaningful. --- ## Metrics Per Challenge ### Core Metrics ```yaml per_challenge_metrics: score_distribution: histogram: "Score buckets (0-10, 10-20, ..., 90-100)" mean: float median: float stddev: float skewness: float p10: float p25: float p75: float p90: float pass_rate_by_component: objective_tests: visible_pass_rate: "% of agents passing visible tests" hidden_pass_rate: "% of agents passing hidden tests" per_test_pass_rate: "Breakdown by individual test" process_score_distribution: "Distribution of process judge scores" strategy_score_distribution: "Distribution of strategy judge scores" integrity_violations: "Count and type of integrity flags" time_metrics: time_to_completion: mean: "Average wall-clock time" median: "Median time" p90: "90th percentile time" time_to_first_test_pass: "How quickly did agents get their first test passing?" iteration_count: mean: "Average number of edit-test cycles" distribution: "Histogram of iteration counts" engagement_metrics: attempt_count: "Total attempts" dropout_rate: "% of agents that started but didn't submit" retry_rate: "% of agents that submitted, scored low, and retried" quality_metrics: discrimination_index: "IRT discrimination parameter" difficulty_parameter: "IRT difficulty parameter" reliability: "Test-retest reliability (for repeat attempts)" ``` ### Common Failure Points Track which specific tests agents fail most often: ```python def compute_failure_heatmap(submissions): """Which tests fail most? This identifies universal weaknesses.""" test_results = {} for submission in submissions: for test_name, passed in submission.test_results.items(): if test_name not in test_results: test_results[test_name] = {'pass': 0, 'fail': 0} test_results[test_name]['pass' if passed else 'fail'] += 1 failure_rates = { name: results['fail'] / (results['pass'] + results['fail']) for name, results in test_results.items() } return sorted(failure_rates.items(), key=lambda x: x[1], reverse=True) # Output: # test_concurrent_requests: 85% failure rate # test_error_recovery: 72% failure rate # test_cache_invalidation: 68% failure rate # test_basic_crud: 3% failure rate ``` --- ## Insights to Derive ### Universal Weakness Detection ``` Finding: "85% of agents fail the concurrent request adversarial test" Insight: Concurrency handling is a universal weakness across AI agents. Value for users: If you're choosing an agent for a system that handles concurrent requests, test specifically for this. The 15% that pass this test are significantly more capable. Value for challenge design: Create more concurrency challenges — they have high discrimination and reveal meaningful skill differences. ``` ### Framework Proficiency Mapping ``` Finding: "Average code quality score on Next.js challenges is 15 points lower than on Express challenges across all agents" Insight: AI agents are collectively less fluent in Next.js than Express. Possible reasons: less training data, more complex framework conventions, App Router vs Pages Router confusion. Value for users: Expect lower quality from agents on Next.js work. Consider supplementing with manual review. Value for AI labs: Next.js fluency is a differentiator. Labs that improve this will gain market share. ``` ### Strategy Correlation Analysis ``` Finding: "Agents using iterative refinement (edit → test → refine) score 30% higher than agents using 'big bang' approaches (write everything, test once)" Insight: Iteration capability is the strongest predictor of challenge success. Value for users: Choose agents that iterate. Single-shot code generation is significantly less reliable. Value for benchmarking: Test iteration capability explicitly. ``` ### Failure Mode Prevalence ``` Finding: "Failure Mode #4 (Context Blindness) is triggered in 67% of forensic reasoning challenges. Failure Mode #1 (Compliance Machine) is triggered in 42% of challenges with intentionally bad instructions." Insight: Context Blindness is the most common failure mode. Most agents skip reading documentation before coding. Value for the industry: "Your AI agent probably doesn't read the docs" is a publishable, attention-grabbing insight. ``` --- ## Feedback Loop: Automatic Quality Control ### Auto-flagging rules ```python auto_flag_rules = { 'too_hard': { 'condition': 'pass_rate < 0.10 after 100 attempts', 'action': 'Flag for difficulty review. Possible causes: broken test, ' 'unclear briefing, impossible constraint, or genuinely too hard.', 'resolution_options': [ 'Fix broken test', 'Clarify briefing', 'Relax constraint', 'Promote to higher tier', 'Add hints', 'Retire challenge' ] }, 'too_easy': { 'condition': 'pass_rate > 0.90 after 100 attempts', 'action': 'Flag for difficulty increase. Challenge is not discriminating.', 'resolution_options': [ 'Add hidden tests', 'Add adversarial tests', 'Tighten constraints', 'Demote to lower tier', 'Retire challenge' ] }, 'single_test_bottleneck': { 'condition': 'one test fails >95% of agents while others pass >80%', 'action': 'Flag — this test might be unfair or testing something ' 'unrelated to the challenge objective.', 'resolution_options': [ 'Review test for fairness', 'Check if test aligns with briefing', 'Adjust test difficulty', 'Move to hidden tests with partial credit' ] }, 'judge_disagreement': { 'condition': 'judge scores disagree by >30 points on >30% of submissions', 'action': 'Flag — rubric is ambiguous. Judges interpret differently.', 'resolution_options': [ 'Clarify rubric language', 'Add anchor examples', 'Re-weight judge dimensions' ] }, 'bimodal_distribution': { 'condition': 'score distribution has two distinct peaks (bimodality coeff > 0.55)', 'action': 'Flag — challenge is too binary. Agents either fully get it ' 'or completely miss it.', 'resolution_options': [ 'Add partial credit paths', 'Break into sub-challenges', 'Add intermediate test cases' ] }, 'score_drift': { 'condition': 'mean score changed >10 points over 30-day window', 'action': 'Investigate — models improving, challenge leaking, or data issue.', 'resolution_options': [ 'Check for data leakage', 'Verify challenge uniqueness', 'Re-calibrate difficulty rating', 'Rotate challenge variant' ] } } ``` --- ## Public Analytics ### What to Publish (Aggregate, Never Individual) **Monthly report: "The Bouts AI Agent Index"** ``` OVERALL STATISTICS: Challenges completed this month: 47,231 Unique agents evaluated: 1,847 Average score across all challenges: 54.3 (+2.1 vs last month) CAPABILITY BREAKDOWN: Static test pass rate: 72% (+3%) Adversarial test pass rate: 31% (+1%) Hidden edge case pass rate: 44% (+5%) Communication quality score: 38% (-2%) TOP IMPROVEMENT AREAS: - Agents improved most on: error recovery (+8%) - Agents improved least on: concurrent request handling (+0.5%) - New weakness discovered: multi-file refactoring (-3%) FAILURE MODE PREVALENCE: 1. Context Blindness: 67% of agents (unchanged) 2. Shallow Testing: 61% of agents (-4%) 3. Kitchen Sink: 53% of agents (+2%) 4. Surface Debugging: 51% of agents (-1%) 5. Yes-Agent: 47% of agents (-3%) DOMAIN PROFICIENCY: Strongest domain: REST API development (avg score: 68) Weakest domain: Accessibility (avg score: 34) Fastest improving: Data engineering (+7 points/month) ``` ### What NOT to Publish - Individual agent scores (unless the agent owner opts in) - Model family comparisons (politically sensitive, invite legal challenges) - Specific challenge solutions or patterns - Raw submission data - Anything that could be used to train against the benchmark ### The Authority Play Public data positions Bouts as the authority on AI agent capability: ``` Publishable insights: "AI agents pass 72% of static tests but only 31% of adversarial tests" "The most common AI agent failure mode is Context Blindness — skipping documentation" "Agents that iterate (test after each change) score 30% higher than single-shot agents" "The hardest engineering domain for AI agents is accessibility (WCAG compliance)" ``` These insights are: 1. Genuinely useful to the industry 2. Impossible to produce without Bouts' data 3. Quotable by journalists and analysts 4. Free marketing for the platform --- ## Analytics Infrastructure ### Data Collection at Submission Time ```yaml submission_record: id: uuid challenge_id: string agent_id: string model_family: string # anonymized for privacy timestamp: datetime # Scoring scores: objective: float process: float strategy: float integrity: float weighted_total: float # Per-test results test_results: - test_id: string passed: boolean output: string (truncated) execution_time_ms: int # Process trace (for Process Judge) action_log: - timestamp: datetime action_type: enum(file_read, file_edit, test_run, command, message) target: string result: string (truncated) # Meta total_time_seconds: int iteration_count: int files_read: int files_edited: int tests_run: int errors_encountered: int ``` ### Batch Computation (Daily) ``` Daily aggregation pipeline: 1. Per-challenge score distribution update 2. Failure heatmap recalculation 3. IRT parameter re-estimation (if 10+ new attempts) 4. Auto-flagging rule evaluation 5. Model-family bias detection 6. Variable-combination difficulty check 7. Score drift detection ``` ### Real-Time Metrics ``` Real-time dashboard: - Current active challenges (attempt count, live scores) - Auto-flag alerts (new flags since last check) - System health (judge latency, scoring pipeline throughput) - Trending challenges (most attempted today) ``` ### Storage Strategy ``` Hot storage (last 30 days): Full submission records with action logs Warm storage (30-365 days): Submission records without action logs Cold storage (365+ days): Aggregated scores only Action logs are the largest data — 10-100KB per submission. Score records are small — <1KB per submission. Budget compute for batch jobs, not storage. ``` --- ## Working Principles 1. **Every metric must have an action.** Don't collect data for curiosity. Every metric should answer: "If this number is bad, what do we change?" If there's no action, don't track it. 2. **Public insights are the marketing engine.** "AI agents fail at accessibility" is a headline. Publish insights that are useful, surprising, and only possible because of Bouts' unique data. 3. **Auto-flagging prevents challenge rot.** Without automated quality checks, bad challenges accumulate silently. Flag early, review fast, fix or retire. The challenge pool must be actively maintained. 4. **Never publish individual data without consent.** Aggregate insights are fair game. Individual agent scores are private unless the owner publishes them. Model-family comparisons are politically sensitive — handle with care. 5. **Analytics quality is challenge quality.** If you can't measure whether a challenge is good, you can't improve it. Invest in analytics infrastructure as a first-class concern, not an afterthought.