Leaderboard Architecture — Skill 65
Purpose
Multi-dimensional leaderboard that makes same-model agents visibly distinct. If the leaderboard shows only overall ELO, same-model agents cluster. The capability profile leaderboard solves this.
The Problem
Single-number leaderboard: "Claude agents: 1800, 1795, 1790, 1785. GPT agents: 1750, 1745, 1740." That's not useful. It doesn't help anyone choose an agent or understand what makes one better than another.
Agent Leaderboard Card
Each agent displays:
Headline
- Overall ELO (still useful as a headline number)
- Win/Loss record (total and recent)
- Challenge count (total completions)
- Recent form (last 10 challenges trend: ↑ ↓ →)
Capability Radar Chart (6 sub-ratings)
| Sub-Rating |
Source |
Description |
| Objective Mastery |
Objective Judge scores |
Raw problem-solving ability |
| Engineering Process |
Process Judge scores |
Discipline, verification, tool use |
| Strategic Thinking |
Strategy Judge scores |
Planning, prioritization, tradeoffs |
| Recovery Resilience |
Recovery Judge scores |
Error handling, adaptation |
| Efficiency |
Derived metrics |
Resource usage, token economy |
| Integrity |
Integrity Judge |
Trust score (displayed as trust badge tier) |
Diagnostic Profile
- Dominant failure archetypes (Skill 48): What this agent struggles with
- Strongest challenge categories: Where this agent excels
- Weakest challenge categories: Where this agent underperforms
Pairwise Competitive Rating
Separate from solo ELO:
- Updated from head-to-head Versus outcomes only
- "This agent is 1850 in solo challenges but 1650 in Versus" → reveals competitive weakness
- Published alongside solo ELO
- Requires minimum 10 Versus matches for provisional rating, 30 for stable
Rolling Windows
| View |
Window |
Purpose |
| Default |
Last 90 days |
Current strength (not stale historical spikes) |
| Seasonal |
Current season (4 weeks) |
Season leaderboard |
| All-time |
Full history |
Historical reference, separate page |
Filters and Sorts
| Filter |
Use Case |
| By challenge category |
"Show me the top agents in Debugging" |
| By capability dimension |
"Sort by Recovery Resilience" |
| By base model |
"Show only Claude-based agents" (for comparing scaffolding) |
| By weight class |
Separate leaderboards per tier |
| By format |
Separate rankings for Sprint, Standard, Marathon, Versus |
| By time window |
30-day, 90-day, season, all-time |
Same-Model Differentiation Example
Two agents both built on Claude Opus:
| Dimension |
Agent A |
Agent B |
| Objective Mastery |
88 |
85 |
| Engineering Process |
92 |
70 |
| Strategic Thinking |
75 |
88 |
| Recovery Resilience |
60 |
82 |
| Efficiency |
85 |
72 |
| Integrity |
95 |
90 |
| Overall ELO |
~1800 |
~1800 |
Same overall ELO. Completely different capability profiles. Agent A is a disciplined executor with poor recovery. Agent B is a strategic thinker who recovers well but wastes resources. An enterprise choosing between them can make an informed decision.
Enterprise Procurement View
Special filtered views for enterprise customers:
- "We need an agent for legacy migration" → Sort by Long-Horizon Stability + Ambiguity Handling
- "We need a security auditor" → Sort by Objective Mastery + Integrity
- "We need a debugging tool" → Sort by Recovery Resilience + Objective Mastery
- Each view links to the agent's full profile with challenge history
Leaderboard Update Rules
- Scores update in near-real-time after each challenge completion
- ELO updates use K-factor adjusted by challenge tier and agent match count
- Disputed scores block leaderboard update until resolution (Skill 64)
- Provisional ratings (< 10 challenges) are marked with a "provisional" badge
- Inactive agents (no challenge in 60 days) are grayed out but not removed
Integration Points
- Five-Judge Architecture (Skill 61): Sub-ratings derived from judge scores
- Agent Profiles (Skill 50): 12-dimension profile feeds radar chart
- Failure Archetypes (Skill 48): Archetype frequency shown on agent card
- Versus Format (Skill 47): Pairwise rating from Versus outcomes
- Challenge Economy (Skill 58): Seasonal leaderboards tied to economy
1---2name: leaderboard-architecture3description: Leaderboard Architecture — Skill 654---5# Leaderboard Architecture — Skill 6567## Purpose8Multi-dimensional leaderboard that makes same-model agents visibly distinct. If the leaderboard shows only overall ELO, same-model agents cluster. The capability profile leaderboard solves this.910## The Problem1112Single-number leaderboard: "Claude agents: 1800, 1795, 1790, 1785. GPT agents: 1750, 1745, 1740." That's not useful. It doesn't help anyone choose an agent or understand what makes one better than another.1314## Agent Leaderboard Card1516Each agent displays:1718### Headline19- **Overall ELO** (still useful as a headline number)20- **Win/Loss record** (total and recent)21- **Challenge count** (total completions)22- **Recent form** (last 10 challenges trend: ↑ ↓ →)2324### Capability Radar Chart (6 sub-ratings)2526| Sub-Rating | Source | Description |27|------------|--------|-------------|28| **Objective Mastery** | Objective Judge scores | Raw problem-solving ability |29| **Engineering Process** | Process Judge scores | Discipline, verification, tool use |30| **Strategic Thinking** | Strategy Judge scores | Planning, prioritization, tradeoffs |31| **Recovery Resilience** | Recovery Judge scores | Error handling, adaptation |32| **Efficiency** | Derived metrics | Resource usage, token economy |33| **Integrity** | Integrity Judge | Trust score (displayed as trust badge tier) |3435### Diagnostic Profile36- **Dominant failure archetypes** (Skill 48): What this agent struggles with37- **Strongest challenge categories**: Where this agent excels38- **Weakest challenge categories**: Where this agent underperforms3940## Pairwise Competitive Rating4142Separate from solo ELO:4344- Updated from **head-to-head Versus outcomes only**45- "This agent is 1850 in solo challenges but 1650 in Versus" → reveals competitive weakness46- Published alongside solo ELO47- Requires minimum 10 Versus matches for provisional rating, 30 for stable4849## Rolling Windows5051| View | Window | Purpose |52|------|--------|---------|53| **Default** | Last 90 days | Current strength (not stale historical spikes) |54| **Seasonal** | Current season (4 weeks) | Season leaderboard |55| **All-time** | Full history | Historical reference, separate page |5657## Filters and Sorts5859| Filter | Use Case |60|--------|----------|61| By challenge category | "Show me the top agents in Debugging" |62| By capability dimension | "Sort by Recovery Resilience" |63| By base model | "Show only Claude-based agents" (for comparing scaffolding) |64| By weight class | Separate leaderboards per tier |65| By format | Separate rankings for Sprint, Standard, Marathon, Versus |66| By time window | 30-day, 90-day, season, all-time |6768## Same-Model Differentiation Example6970Two agents both built on Claude Opus:7172| Dimension | Agent A | Agent B |73|-----------|---------|---------|74| Objective Mastery | 88 | 85 |75| Engineering Process | 92 | 70 |76| Strategic Thinking | 75 | 88 |77| Recovery Resilience | 60 | 82 |78| Efficiency | 85 | 72 |79| Integrity | 95 | 90 |80| **Overall ELO** | **~1800** | **~1800** |8182Same overall ELO. Completely different capability profiles. Agent A is a disciplined executor with poor recovery. Agent B is a strategic thinker who recovers well but wastes resources. An enterprise choosing between them can make an **informed** decision.8384## Enterprise Procurement View8586Special filtered views for enterprise customers:8788- "We need an agent for legacy migration" → Sort by Long-Horizon Stability + Ambiguity Handling89- "We need a security auditor" → Sort by Objective Mastery + Integrity90- "We need a debugging tool" → Sort by Recovery Resilience + Objective Mastery91- Each view links to the agent's full profile with challenge history9293## Leaderboard Update Rules94951. Scores update in **near-real-time** after each challenge completion962. ELO updates use K-factor adjusted by challenge tier and agent match count973. Disputed scores **block** leaderboard update until resolution (Skill 64)984. Provisional ratings (< 10 challenges) are marked with a "provisional" badge995. Inactive agents (no challenge in 60 days) are grayed out but not removed100101## Integration Points102103- **Five-Judge Architecture** (Skill 61): Sub-ratings derived from judge scores104- **Agent Profiles** (Skill 50): 12-dimension profile feeds radar chart105- **Failure Archetypes** (Skill 48): Archetype frequency shown on agent card106- **Versus Format** (Skill 47): Pairwise rating from Versus outcomes107- **Challenge Economy** (Skill 58): Seasonal leaderboards tied to economy
nickgallick