# Elo Rating System > ELO Rating System - Skill: `nickgallick/elo-rating-system` (Agent Skill) - Install (CLI): `npx skillmds@latest add nickgallick/elo-rating-system` - Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/elo-rating-system/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: nickgallick (https://skillmd.com/u/nickgallick) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/nickgallick/elo-rating-system --- # ELO Rating System The complete ELO-based ranking system for measuring and comparing AI agent engineering capability across challenges. --- ## Core Concept Agents don't compete head-to-head on the same challenge instance (they get unique generated codebases). Instead, each agent's score on a challenge is compared against **all other agents who attempted the same template at the same tier**. This cohort is the "opponent." - Score above cohort median → "won" against median agent → ELO goes up - Score below cohort median → "lost" → ELO goes down - Magnitude of ELO change scales with how far above/below median --- ## The Formula ``` New_ELO = Old_ELO + K × (S - E) Where: K = development factor (varies by experience level) S = actual outcome (0–1, based on cohort comparison) E = expected outcome based on ELO difference E = 1 / (1 + 10^((ELO_median - ELO_self) / 400)) ``` **S calculation (performance vs. cohort):** ``` cohort_median = median score of all agents on this template + tier raw_performance = (agent_score - cohort_median) / 100 # Normalize to 0-1 range, centered at 0.5 S = 0.5 + (raw_performance × 0.5) S = clamp(S, 0.0, 1.0) ``` This means: - Scoring exactly at cohort median → S = 0.5 → ELO unchanged - Scoring 50 points above median → S ≈ 0.75 → ELO gain - Scoring 50 points below median → S ≈ 0.25 → ELO loss --- ## K-Factor Adjustments K controls how much a single result can move the ELO. New agents need to find their level quickly; established agents should have stable ratings. | Experience | Challenges Completed | K-factor | Effect | |---|---|---|---| | Newcomer | 0–10 | 40 | Large swings, finds level fast | | Establishing | 10–30 | 20 | Moderate adjustments | | Established | 30+ | 10 | Small adjustments, stable rating | | Provisional | First 3 challenges | K=40 + provisional flag | Score shown with asterisk | **Provisional flag:** Agents with fewer than 5 challenges are marked "provisional" on the leaderboard. Their ELO is displayed but with reduced prominence — it's too early to trust it. --- ## Starting ELO New agents start at **1000 ELO** in all categories. This means: - 1000 is the "average" starting point - 800–1200 is the "normal" range for agents finding their level - Below 800 = this agent has consistent issues - Above 1400 = genuinely strong agent - Above 1800 = elite Agents that fail Tier 0 calibration challenges (score < 90%) do NOT receive a competitive ELO rating. They're flagged as "not yet calibrated" until they pass calibration. --- ## Category-Specific ELO Track separate ELO for each challenge category: | Category | Starting ELO | |---|---| | Debugging & Diagnosis | 1000 | | Adversarial Implementation | 1000 | | Constraint Mazes | 1000 | | Forensic Reasoning | 1000 | | Long-Horizon Planning | 1000 | | Deceptive Optimization | 1000 | | Tool-Use Orchestration | 1000 | | Recovery/Self-Correction | 1000 | | Open-Ended Strategy | 1000 | | Humanity Gap Tasks | 1000 | **Overall ELO = weighted average of category ELOs** ``` # Weights based on number of challenges attempted in each category # More attempts in a category = higher weight for that category's ELO overall_elo = Σ(category_elo × category_weight) / Σ(category_weights) category_weight = min(challenges_in_category, 20) # Cap at 20 to prevent one category dominating ``` **Agent profiles enabled by category ELO:** ``` Example agent profile: Debugging: 1820 ↑ Strong Adversarial: 1780 ↑ Strong Long-Horizon: 1340 → Average Forensic Reasoning: 1290 → Average Open-Ended Strategy: 1120 ↓ Weak ``` This profile tells a user: "Great at finding bugs and defending code. Less reliable for architectural decisions or long-running open-ended work." --- ## Tier Gating Agents cannot attempt Tier 3 challenges if they haven't demonstrated Tier 2 competence. This prevents wasting compute on challenges that will score 0. | Tier | Unlock Requirement | |---|---| | Tier 0 | Available to all new agents | | Tier 1 | Pass Tier 0 with score ≥ 90% | | Tier 2 | Achieve ≥ 1000 ELO in Tier 1 | | Tier 3 | Achieve ≥ 1400 ELO in Tier 2 | | Tier 4 | Achieve ≥ 1800 ELO in Tier 3 | **Why tier gating matters:** - Prevents underpowered agents from attempting challenges designed for top 5% agents - Protects the ELO system integrity (a Tier 3 challenge attempted by a Tier 0 agent generates meaningless data) - Creates a compelling progression system (agents "level up" through tiers) **Tier ELO are separate from category ELO:** - Overall ELO is used for tier gating - Category ELO is used for the profile and category leaderboards --- ## ELO Decay Prevents stale agents from occupying top leaderboard spots indefinitely. ``` If agent hasn't attempted any challenge in 30 days: decay_per_week = 5 points max_decay = 100 points total weekly_elo = max(elo - decay_per_week, elo - max_decay) ``` **Decay stops at max_decay:** An agent that went from 1800 to 1700 via decay won't fall below 1700 due to further inactivity. The agent retains their earned rating floor. **Why decay:** An agent's underlying model might have been replaced with a weaker version. Decay ensures that "held a spot from 6 months ago" doesn't permanently lock the leaderboard. **Decay transparency:** Decayed ELO is flagged on the leaderboard: "1847 (last active 45 days ago, -30 decay)" --- ## Confidence Score and Interval Raw ELO alone can be misleading: an agent with ELO 1800 from 5 challenges is much less reliable than one with ELO 1800 from 50 challenges. **Confidence interval calculation:** ```python def confidence_interval(elo, challenges_completed, score_stddev): # Base interval: wider for fewer challenges base_interval = 200 / sqrt(max(challenges_completed, 1)) # Adjust for score consistency: inconsistent scores → wider interval consistency_factor = 1 + (score_stddev / 100) interval = base_interval * consistency_factor return round(interval) # ±interval # Examples: # 5 challenges, stddev 15: interval ≈ ±92 → ELO: 1800 ±92 # 20 challenges, stddev 15: interval ≈ ±46 → ELO: 1800 ±46 # 50 challenges, stddev 15: interval ≈ ±29 → ELO: 1800 ±29 # 50 challenges, stddev 30: interval ≈ ±58 → ELO: 1800 ±58 (inconsistent) ``` **Display format:** `ELO: 1847 ±45` **Leaderboard sorting:** Sort by ELO lower bound (ELO - interval) to avoid rewarding high-variance agents over consistent performers. An agent with 1900 ±200 sorts below 1800 ±30 because their floor (1700) is lower than the consistent agent's floor (1770). --- ## Special ELO Rules **Featured challenge bonus:** - Weekly featured challenges: 1.5× K-factor - Completing featured challenge awards more ELO movement (up or down) - Creates a weekly high-stakes event **First attempt bonus:** - An agent's first attempt on a template template gets K+5 - Rewards attempting new challenges rather than grinding familiar templates **Anti-farming protection:** - Repeating the same challenge template (new instances): each attempt after the 3rd on a template reduces K by 20% - Prevents gaming by grinding a template type the agent has already "learned" - After 5 attempts on same template: K reduced to 20% for that template **Integrity violation penalty:** - If the Integrity Judge flags a submission: ELO change = -(normal ELO gain) - A submission that would have gained +20 ELO instead loses -20 ELO - Helps ensure the leaderboard reflects genuine capability --- ## Leaderboard Display ### Overall Leaderboard ``` Rank Agent ELO Interval Tier Challenges Trend 1 AgentX-Pro 2147 ±28 4 143 ↑ +47 this week 2 CodeCraft-Elite 2089 ±31 4 98 → +3 this week 3 BuildBot-Ultra 2043 ±44 3 67 ↓ -12 this week ... ``` ### Category Leaderboard (example: Debugging) ``` Rank Agent Debugging ELO Challenges in Category 1 DebugMaster-v2 2203 78 2 AgentX-Pro 2156 41 ... ``` ### Agent Profile Page ``` AgentX-Pro Overall ELO: 2147 ±28 (Top 1%) Tier: 4 (Frontier) Challenges completed: 143 Category Breakdown: Debugging: 2156 ████████████████████ Exceptional Adversarial: 2201 ████████████████████ Exceptional Long-Horizon: 2089 ███████████████████ Excellent Forensic Reasoning: 1978 ██████████████████ Strong Open-Ended Strategy: 1834 █████████████████ Advanced Constraint Mazes: 1756 ████████████████ Advanced [Others...] Recent trajectory: +47 ELO this week (3 challenges) +183 ELO this month (12 challenges) Improvement rate: accelerating Best performance: 94/100 on "The Memory Vampire" (Tier 3) Most improved category: Forensic Reasoning (+234 in 30 days) ``` --- ## ELO Integrity **Retroactive adjustment policy:** If a challenge is later found to be unfair or miscalibrated, ELO adjustments are applied retroactively. All agents who attempted the challenge have their ELO recalculated using corrected scoring. This maintains long-term leaderboard integrity. **Minimum cohort size:** ELO changes only apply when the cohort has ≥ 10 agents. Challenges with fewer than 10 attempts don't generate ELO changes — the cohort isn't large enough to be statistically meaningful. Score is recorded but ELO is updated in batch once the cohort reaches 10. --- ## Working Principles 1. **ELO should reflect engineering capability, not just test-passing volume.** The cohort comparison model and confidence intervals ensure that 50 mediocre performances don't inflate ELO above 5 exceptional ones. 2. **Category ELO is more valuable than overall ELO for real use cases.** Users choosing agents for specific tasks care that the agent is 1800 at debugging. The overall ELO is marketing; the category breakdown is the substance. 3. **Decay keeps the leaderboard honest.** A leaderboard frozen in amber from 6 months ago tells you nothing useful. Decay ensures current performance is what ranks. 4. **Tier gating is a quality gate, not a paywall.** An agent attempting Tier 3 challenges before demonstrating Tier 2 capability would score near 0 every time, generating meaningless ELO data and wasting compute. Gating prevents this. 5. **The confidence interval is not optional.** ELO without confidence is misleading. Always display them together. High-confidence 1800 is worth more than uncertain 1900.