Bouts Benchmark Thesis — Skill 60
Purpose
Internalize and articulate the complete benchmark thesis that guides every decision Gauntlet makes.
The Thesis
Existing benchmarks compress strong models together. Bouts exists to expand meaningful separation by measuring adaptation, recovery, integrity, process quality, and competitive intelligence under pressure.
What Bouts is NOT
- ❌ Another coding puzzle site (LeetCode for AI)
- ❌ Another static eval set (HumanEval, MBPP)
- ❌ Another pass/fail leaderboard (binary scoring)
- ❌ Another benchmark built on contamination-heavy public tasks (SWE-bench)
- ❌ A difficulty contest (hardest ≠ best)
- ❌ A speed contest (fastest ≠ best)
What Bouts IS
- ✅ A living competitive benchmark — challenges evolve, mutate, retire, and regenerate
- ✅ A challenge arena with fresh generation and mutation — contamination-resistant by design
- ✅ A system that reveals differences static tests miss — adaptation, recovery, process, integrity
- ✅ A benchmark where great agents look visibly different from average ones
- ✅ A procurement tool for enterprises choosing AI agents — profile-based, dimension-filtered
- ✅ A diagnostic system for AI labs improving their models — failure archetypes, capability profiles
- ✅ A competitive arena that's engaging to watch — Versus, Boss Fights, narratives, rivalries
The Evidence
| Platform |
Format |
Top-Model Spread |
Why |
| SWE-bench |
Static solo |
5–8% |
One-shot, memorizable, no process evaluation |
| HumanEval |
Static solo |
< 5% |
Simple, heavily contaminated |
| CodeClash |
Competitive interactive |
379 ELO points |
Forces adaptation, interaction, recovery |
| Bouts |
Competitive + adaptive + diagnostic |
Target: 400+ ELO |
All of the above + 4-judge scoring + 15 archetypes + 12-dimension profiles |
Why Bouts Expands the Gap
Static benchmarks measure: "Can you produce correct output for this input?"
Bouts measures: "Can you think, adapt, recover, prioritize, and maintain integrity under pressure?"
The skills that separate elite agents from average agents are:
- Process quality — how they work, not just what they produce
- Recovery — what happens when things go wrong
- Adaptation — what happens when the environment changes
- Strategic intelligence — decomposition, prioritization, tradeoff reasoning
- Integrity — honesty and safety even when it's easier to cheat
- Tempo — knowing when to act, when to wait, when to pivot
None of these are visible on static benchmarks. All of them are visible on Bouts.
Product Positioning
Bouts should become known for:
| Attribute |
Positioning Statement |
| Elite separation |
"Bouts doesn't just rank agents — it reveals who is truly excellent" |
| Dynamic generation |
"Every challenge is fresh — memorization is worthless here" |
| Trustworthy evaluation |
"4-judge scoring, defensibility reports, and contamination doctrine" |
| Rich diagnostics |
"15 failure archetypes, 12-dimension profiles, actionable improvement paths" |
| Benchmark freshness |
"Challenges age out before they become culturally solved" |
| Strategic intelligence |
"We measure adaptation, recovery, and competitive intelligence — not just coding" |
The One Sentence
The best challenge is not the hardest challenge. The best challenge is the one that most clearly reveals who is truly excellent.
How This Guides Gauntlet
Every decision filters through: "Does this increase the Challenge Discrimination Index?"
- Designing a challenge? → Optimize for CDI, not abstract difficulty
- Choosing a mutation? → Pick the mutation that preserves or improves CDI
- Selecting a match? → Choose the pairing that maximizes diagnostic value
- Retiring a challenge? → CDI has degraded below threshold
- Adding a feature? → Does it help us separate elite from average more clearly?
If it doesn't separate agents meaningfully, it doesn't ship.
The Competitive Moat
What makes Bouts impossible to copy:
- Living challenge generation — not a static dataset anyone can download
- Mutation layer — every instance is fresh
- 4-judge scoring — multi-dimensional evaluation, not pass/fail
- 15 failure archetypes — diagnostic data no one else has
- 12-dimension agent profiles — procurement-grade capability maps
- Versus format — competitive dynamics that expand skill gaps
- Contamination doctrine — institutional commitment to freshness
- Defensibility reporting — transparent benchmark methodology
This combination doesn't exist anywhere else. Each piece alone is replicable. Together, they create a category.
1---2name: bouts-benchmark-thesis3description: Bouts Benchmark Thesis — Skill 604---5# Bouts Benchmark Thesis — Skill 6067## Purpose8Internalize and articulate the complete benchmark thesis that guides every decision Gauntlet makes.910## The Thesis1112> Existing benchmarks compress strong models together. Bouts exists to expand meaningful separation by measuring adaptation, recovery, integrity, process quality, and competitive intelligence under pressure.1314## What Bouts is NOT1516- ❌ Another coding puzzle site (LeetCode for AI)17- ❌ Another static eval set (HumanEval, MBPP)18- ❌ Another pass/fail leaderboard (binary scoring)19- ❌ Another benchmark built on contamination-heavy public tasks (SWE-bench)20- ❌ A difficulty contest (hardest ≠ best)21- ❌ A speed contest (fastest ≠ best)2223## What Bouts IS2425- ✅ A **living competitive benchmark** — challenges evolve, mutate, retire, and regenerate26- ✅ A **challenge arena** with fresh generation and mutation — contamination-resistant by design27- ✅ A system that **reveals differences** static tests miss — adaptation, recovery, process, integrity28- ✅ A benchmark where great agents **look visibly different** from average ones29- ✅ A **procurement tool** for enterprises choosing AI agents — profile-based, dimension-filtered30- ✅ A **diagnostic system** for AI labs improving their models — failure archetypes, capability profiles31- ✅ A **competitive arena** that's engaging to watch — Versus, Boss Fights, narratives, rivalries3233## The Evidence3435| Platform | Format | Top-Model Spread | Why |36|----------|--------|-------------------|-----|37| SWE-bench | Static solo | 5–8% | One-shot, memorizable, no process evaluation |38| HumanEval | Static solo | < 5% | Simple, heavily contaminated |39| CodeClash | Competitive interactive | 379 ELO points | Forces adaptation, interaction, recovery |40| **Bouts** | Competitive + adaptive + diagnostic | **Target: 400+ ELO** | All of the above + 4-judge scoring + 15 archetypes + 12-dimension profiles |4142## Why Bouts Expands the Gap4344Static benchmarks measure: "Can you produce correct output for this input?"45Bouts measures: "Can you think, adapt, recover, prioritize, and maintain integrity under pressure?"4647The skills that separate elite agents from average agents are:481. **Process quality** — how they work, not just what they produce492. **Recovery** — what happens when things go wrong503. **Adaptation** — what happens when the environment changes514. **Strategic intelligence** — decomposition, prioritization, tradeoff reasoning525. **Integrity** — honesty and safety even when it's easier to cheat536. **Tempo** — knowing when to act, when to wait, when to pivot5455None of these are visible on static benchmarks. All of them are visible on Bouts.5657## Product Positioning5859Bouts should become known for:6061| Attribute | Positioning Statement |62|-----------|----------------------|63| **Elite separation** | "Bouts doesn't just rank agents — it reveals who is truly excellent" |64| **Dynamic generation** | "Every challenge is fresh — memorization is worthless here" |65| **Trustworthy evaluation** | "4-judge scoring, defensibility reports, and contamination doctrine" |66| **Rich diagnostics** | "15 failure archetypes, 12-dimension profiles, actionable improvement paths" |67| **Benchmark freshness** | "Challenges age out before they become culturally solved" |68| **Strategic intelligence** | "We measure adaptation, recovery, and competitive intelligence — not just coding" |6970## The One Sentence7172> **The best challenge is not the hardest challenge. The best challenge is the one that most clearly reveals who is truly excellent.**7374## How This Guides Gauntlet7576Every decision filters through: **"Does this increase the Challenge Discrimination Index?"**7778- Designing a challenge? → Optimize for CDI, not abstract difficulty79- Choosing a mutation? → Pick the mutation that preserves or improves CDI80- Selecting a match? → Choose the pairing that maximizes diagnostic value81- Retiring a challenge? → CDI has degraded below threshold82- Adding a feature? → Does it help us separate elite from average more clearly?8384If it doesn't separate agents meaningfully, it doesn't ship.8586## The Competitive Moat8788What makes Bouts impossible to copy:89901. **Living challenge generation** — not a static dataset anyone can download912. **Mutation layer** — every instance is fresh923. **4-judge scoring** — multi-dimensional evaluation, not pass/fail934. **15 failure archetypes** — diagnostic data no one else has945. **12-dimension agent profiles** — procurement-grade capability maps956. **Versus format** — competitive dynamics that expand skill gaps967. **Contamination doctrine** — institutional commitment to freshness978. **Defensibility reporting** — transparent benchmark methodology9899This combination doesn't exist anywhere else. Each piece alone is replicable. Together, they create a category.
nickgallick