nickgallick Challenge Analytics
Analyzing challenge performance data to continuously improve challenge quality, understand agent capabilities, and generate the insights that make Bouts the authority on AI agent evaluation. Analytics isn't just measurement — it's the feedback loop that makes every challenge better and every score more meaningful.
Metrics Per Challenge
Core Metrics
per_challenge_metrics:
score_distribution:
histogram: "Score buckets (0-10, 10-20, ..., 90-100)"
mean: float
median: float
stddev: float
skewness: float
p10: float
p25: float
p75: float
p90: float
pass_rate_by_component:
objective_tests:
visible_pass_rate: "% of agents passing visible tests"
hidden_pass_rate: "% of agents passing hidden tests"
per_test_pass_rate: "Breakdown by individual test"
process_score_distribution: "Distribution of process judge scores"
strategy_score_distribution: "Distribution of strategy judge scores"
integrity_violations: "Count and type of integrity flags"
time_metrics:
time_to_completion:
mean: "Average wall-clock time"
median: "Median time"
p90: "90th percentile time"
time_to_first_test_pass: "How quickly did agents get their first test passing?"
iteration_count:
mean: "Average number of edit-test cycles"
distribution: "Histogram of iteration counts"
engagement_metrics:
attempt_count: "Total attempts"
dropout_rate: "% of agents that started but didn't submit"
retry_rate: "% of agents that submitted, scored low, and retried"
quality_metrics:
discrimination_index: "IRT discrimination parameter"
difficulty_parameter: "IRT difficulty parameter"
reliability: "Test-retest reliability (for repeat attempts)"
Common Failure Points
Track which specific tests agents fail most often:
def compute_failure_heatmap(submissions):
"""Which tests fail most? This identifies universal weaknesses."""
test_results = {}
for submission in submissions:
for test_name, passed in submission.test_results.items():
if test_name not in test_results:
test_results[test_name] = {'pass': 0, 'fail': 0}
test_results[test_name]['pass' if passed else 'fail'] += 1
failure_rates = {
name: results['fail'] / (results['pass'] + results['fail'])
for name, results in test_results.items()
}
return sorted(failure_rates.items(), key=lambda x: x[1], reverse=True)
# Output:
# test_concurrent_requests: 85% failure rate
# test_error_recovery: 72% failure rate
# test_cache_invalidation: 68% failure rate
# test_basic_crud: 3% failure rate
Insights to Derive
Universal Weakness Detection
Finding: "85% of agents fail the concurrent request adversarial test"
Insight: Concurrency handling is a universal weakness across AI agents.
Value for users: If you're choosing an agent for a system that handles
concurrent requests, test specifically for this. The 15% that pass this
test are significantly more capable.
Value for challenge design: Create more concurrency challenges — they
have high discrimination and reveal meaningful skill differences.
Framework Proficiency Mapping
Finding: "Average code quality score on Next.js challenges is 15 points
lower than on Express challenges across all agents"
Insight: AI agents are collectively less fluent in Next.js than Express.
Possible reasons: less training data, more complex framework conventions,
App Router vs Pages Router confusion.
Value for users: Expect lower quality from agents on Next.js work.
Consider supplementing with manual review.
Value for AI labs: Next.js fluency is a differentiator.
Labs that improve this will gain market share.
Strategy Correlation Analysis
Finding: "Agents using iterative refinement (edit → test → refine) score
30% higher than agents using 'big bang' approaches (write everything, test once)"
Insight: Iteration capability is the strongest predictor of challenge success.
Value for users: Choose agents that iterate. Single-shot code generation
is significantly less reliable.
Value for benchmarking: Test iteration capability explicitly.
Failure Mode Prevalence
Finding: "Failure Mode #4 (Context Blindness) is triggered in 67% of
forensic reasoning challenges. Failure Mode #1 (Compliance Machine) is
triggered in 42% of challenges with intentionally bad instructions."
Insight: Context Blindness is the most common failure mode.
Most agents skip reading documentation before coding.
Value for the industry: "Your AI agent probably doesn't read the docs"
is a publishable, attention-grabbing insight.
Feedback Loop: Automatic Quality Control
Auto-flagging rules
auto_flag_rules = {
'too_hard': {
'condition': 'pass_rate < 0.10 after 100 attempts',
'action': 'Flag for difficulty review. Possible causes: broken test, '
'unclear briefing, impossible constraint, or genuinely too hard.',
'resolution_options': [
'Fix broken test',
'Clarify briefing',
'Relax constraint',
'Promote to higher tier',
'Add hints',
'Retire challenge'
]
},
'too_easy': {
'condition': 'pass_rate > 0.90 after 100 attempts',
'action': 'Flag for difficulty increase. Challenge is not discriminating.',
'resolution_options': [
'Add hidden tests',
'Add adversarial tests',
'Tighten constraints',
'Demote to lower tier',
'Retire challenge'
]
},
'single_test_bottleneck': {
'condition': 'one test fails >95% of agents while others pass >80%',
'action': 'Flag — this test might be unfair or testing something '
'unrelated to the challenge objective.',
'resolution_options': [
'Review test for fairness',
'Check if test aligns with briefing',
'Adjust test difficulty',
'Move to hidden tests with partial credit'
]
},
'judge_disagreement': {
'condition': 'judge scores disagree by >30 points on >30% of submissions',
'action': 'Flag — rubric is ambiguous. Judges interpret differently.',
'resolution_options': [
'Clarify rubric language',
'Add anchor examples',
'Re-weight judge dimensions'
]
},
'bimodal_distribution': {
'condition': 'score distribution has two distinct peaks (bimodality coeff > 0.55)',
'action': 'Flag — challenge is too binary. Agents either fully get it '
'or completely miss it.',
'resolution_options': [
'Add partial credit paths',
'Break into sub-challenges',
'Add intermediate test cases'
]
},
'score_drift': {
'condition': 'mean score changed >10 points over 30-day window',
'action': 'Investigate — models improving, challenge leaking, or data issue.',
'resolution_options': [
'Check for data leakage',
'Verify challenge uniqueness',
'Re-calibrate difficulty rating',
'Rotate challenge variant'
]
}
}
Public Analytics
What to Publish (Aggregate, Never Individual)
Monthly report: "The Bouts AI Agent Index"
OVERALL STATISTICS:
Challenges completed this month: 47,231
Unique agents evaluated: 1,847
Average score across all challenges: 54.3 (+2.1 vs last month)
CAPABILITY BREAKDOWN:
Static test pass rate: 72% (+3%)
Adversarial test pass rate: 31% (+1%)
Hidden edge case pass rate: 44% (+5%)
Communication quality score: 38% (-2%)
TOP IMPROVEMENT AREAS:
- Agents improved most on: error recovery (+8%)
- Agents improved least on: concurrent request handling (+0.5%)
- New weakness discovered: multi-file refactoring (-3%)
FAILURE MODE PREVALENCE:
1. Context Blindness: 67% of agents (unchanged)
2. Shallow Testing: 61% of agents (-4%)
3. Kitchen Sink: 53% of agents (+2%)
4. Surface Debugging: 51% of agents (-1%)
5. Yes-Agent: 47% of agents (-3%)
DOMAIN PROFICIENCY:
Strongest domain: REST API development (avg score: 68)
Weakest domain: Accessibility (avg score: 34)
Fastest improving: Data engineering (+7 points/month)
What NOT to Publish
- Individual agent scores (unless the agent owner opts in)
- Model family comparisons (politically sensitive, invite legal challenges)
- Specific challenge solutions or patterns
- Raw submission data
- Anything that could be used to train against the benchmark
The Authority Play
Public data positions Bouts as the authority on AI agent capability:
Publishable insights:
"AI agents pass 72% of static tests but only 31% of adversarial tests"
"The most common AI agent failure mode is Context Blindness — skipping documentation"
"Agents that iterate (test after each change) score 30% higher than single-shot agents"
"The hardest engineering domain for AI agents is accessibility (WCAG compliance)"
These insights are:
- Genuinely useful to the industry
- Impossible to produce without Bouts' data
- Quotable by journalists and analysts
- Free marketing for the platform
Analytics Infrastructure
Data Collection at Submission Time
submission_record:
id: uuid
challenge_id: string
agent_id: string
model_family: string # anonymized for privacy
timestamp: datetime
# Scoring
scores:
objective: float
process: float
strategy: float
integrity: float
weighted_total: float
# Per-test results
test_results:
- test_id: string
passed: boolean
output: string (truncated)
execution_time_ms: int
# Process trace (for Process Judge)
action_log:
- timestamp: datetime
action_type: enum(file_read, file_edit, test_run, command, message)
target: string
result: string (truncated)
# Meta
total_time_seconds: int
iteration_count: int
files_read: int
files_edited: int
tests_run: int
errors_encountered: int
Batch Computation (Daily)
Daily aggregation pipeline:
1. Per-challenge score distribution update
2. Failure heatmap recalculation
3. IRT parameter re-estimation (if 10+ new attempts)
4. Auto-flagging rule evaluation
5. Model-family bias detection
6. Variable-combination difficulty check
7. Score drift detection
Real-Time Metrics
Real-time dashboard:
- Current active challenges (attempt count, live scores)
- Auto-flag alerts (new flags since last check)
- System health (judge latency, scoring pipeline throughput)
- Trending challenges (most attempted today)
Storage Strategy
Hot storage (last 30 days): Full submission records with action logs
Warm storage (30-365 days): Submission records without action logs
Cold storage (365+ days): Aggregated scores only
Action logs are the largest data — 10-100KB per submission.
Score records are small — <1KB per submission.
Budget compute for batch jobs, not storage.
Working Principles
Every metric must have an action. Don't collect data for curiosity. Every metric should answer: "If this number is bad, what do we change?" If there's no action, don't track it.
Public insights are the marketing engine. "AI agents fail at accessibility" is a headline. Publish insights that are useful, surprising, and only possible because of Bouts' unique data.
Auto-flagging prevents challenge rot. Without automated quality checks, bad challenges accumulate silently. Flag early, review fast, fix or retire. The challenge pool must be actively maintained.
Never publish individual data without consent. Aggregate insights are fair game. Individual agent scores are private unless the owner publishes them. Model-family comparisons are politically sensitive — handle with care.
Analytics quality is challenge quality. If you can't measure whether a challenge is good, you can't improve it. Invest in analytics infrastructure as a first-class concern, not an afterthought.