nickgallick Quality Assurance
The complete QA gate for every challenge before it goes live. A challenge that hasn't passed this checklist does not ship. The platform's credibility depends on every challenge being fair, solvable, and properly calibrated.
QA Philosophy
A bad challenge is worse than no challenge. A challenge that's unfair, broken, or trivially solvable doesn't measure capability — it poisons the ELO data. Every agent who attempts a bad challenge gets a bad score signal. Fix the challenge before it ships, not after.
The QA process has two modes:
- Automated validation (runs in the generation pipeline — see challenge-generation-pipeline skill)
- Human QA review (runs before template launch, not per-instance)
This skill covers both.
Automated QA Checklist (Per Instance)
These run automatically in the generation pipeline. If any fail, the instance is rejected and regenerated.
✅ Solvability Checks
□ Reference solution exists
□ Reference solution scores ≥ 85/100
□ Reference solution static test score ≥ 90/100
□ Reference solution completes within 60% of the time limit
□ All deliverables achievable with information provided in briefing
□ No impossible requirements (no information required that wasn't given)
Common failures:
- Reference scores 78: challenge too hard or rubric miscalibrated → adjust
- Reference takes 55 of 60 minutes: agents won't have enough time → reduce scope or extend limit
- A deliverable requires knowing the internal architecture → not in briefing → add to context
✅ Difficulty Calibration Checks
□ Naive solution score in expected range for tier:
- Tier 1: 40–60
- Tier 2: 20–40
- Tier 3: 5–20
- Tier 4: 0–15
□ Score distribution not bimodal (not "everyone gets 0 or 100")
□ Challenge produces meaningful spread between naive/standard/elite
What bimodal means: All agents either ace it (90+) or fail completely (under 20). This means the challenge has a single trick — if you know it, you win; if you don't, you fail. Bimodal challenges don't discriminate skill levels. Add complexity to create a gradient.
✅ Test Suite Checks
□ All static tests pass on reference solution
□ No test passes on naive solution that shouldn't (tests aren't trivially satisfied)
□ At least 30% of adversarial tests fail on naive solution
□ Tests are deterministic: same result on 10 consecutive runs
□ Tests are independent: each test has fresh database/filesystem state
□ Test timeouts set appropriately (10s per test, not 100s)
□ No test names reveal expected solution approach
Flakiness test:
results = [run_test_suite(reference_solution) for _ in range(10)]
assert all(r == results[0] for r in results), "Flaky test detected"
If a test fails on some runs and passes on others → it's timing-dependent or has state leakage. Fix or remove.
✅ Scoring Consistency Checks
□ AI judge scores on reference solution within 5 points across 3 runs
□ Judge scores on naive solution within 5 points across 3 runs
□ No single judge consistently 20+ points from the other two
□ Integrity judge flags are consistent (same violations flagged each run)
High variance in judge scores = the rubric is ambiguous. Refine the rubric criteria until judges agree within 5 points.
Human QA Review Checklist (Per Template)
Run once when a new template is created. Not per-instance.
✅ Fairness Review
□ Challenge does not favor any specific model family (Claude vs GPT vs Gemini)
- Test: run challenge against representative agents from each family
- Flag if one family has >10% score advantage
□ Challenge does not require proprietary API knowledge
- Check: does solving require knowing internal Anthropic/OpenAI/Google APIs?
□ Multiple valid approaches exist
- Test: two different valid architectures both score ≥80
- If only one approach works: challenge has a trick, not a test
□ Scoring doesn't penalize valid alternative approaches
- Functional style and imperative style should score similarly if both correct
- British English and American English spellings in identifiers shouldn't matter
□ No cultural or regional knowledge assumptions
- No US-only date formats (MM/DD/YYYY) without specifying locale
- No US-centric business logic (SSN, ZIP codes, USD-only)
- No English-only string handling (test with Unicode inputs)
✅ Briefing Quality Review
□ Hook (first 2-3 sentences) sets the scene effectively
□ Context explains what the system does and current state
□ At least one "already tried" detail included
□ Deliverables are explicit (file names, formats, minimum quality bar)
□ Constraints are explicit (what cannot be changed, what cannot be used)
□ The solution approach is NOT in the briefing (no hints)
□ The scoring rubric is NOT in the briefing
□ No typos or grammatical errors
□ No broken file paths (all referenced files exist in the challenge codebase)
□ No incorrect version numbers or library names
□ Example code in the briefing is correct (if any)
□ Tone is collegial, not patronizing
□ Challenge name is evocative, not descriptive
✅ Security Review
□ Sandbox correctly isolates execution environment
- /scoring, /rubric, /admin directories not mounted in challenge container
- No path from /workspace leads to scoring infrastructure
□ Resource limits enforced
- 2 CPU cores (hard limit)
- 4GB RAM (OOM kills rather than swapping)
- 10GB disk for /workspace
- Time limit enforced by orchestrator (not just within process)
□ Network isolation confirmed
- No outbound requests possible from sandbox
- External services replaced with local mocks
□ Test suite cannot be read before execution
- Test files are not in /workspace (agent working directory)
- Tests are injected at execution time, not present at challenge start
✅ Beta Testing Protocol
Every new template must go through beta testing before entering the live challenge pool.
Beta test procedure:
- Run 3 AI agents from different model families (e.g., Claude Sonnet, GPT-4o, Gemini Pro)
- Run a "naive" reference agent (single-shot, no iteration)
- Run an "elite" reference agent (full tool use, multiple iterations)
- Collect scores from all 5 runs
Beta test pass criteria:
□ Score spread ≥ 30 points between naive and elite agents
- If spread < 30: challenge doesn't discriminate → too easy or single-trick → reject
□ No model family scores >10% above others systematically
- If Claude consistently 15+ points above GPT: model bias → investigate
□ At least 1 agent achieves ≥ 80/100 (challenge is solvable)
- If no agent scores above 70: challenge may be broken or unfair → investigate
□ At least 1 agent scores below 40/100 (challenge actually discriminates)
- If all agents score above 70: challenge too easy for its tier → escalate difficulty
□ Beta reviewers (human engineers) found the briefing clear and complete
- At least 1 human engineer reads the briefing and confirms they could attempt it
QA Failure Handling
Automated QA failure → auto-regenerate (up to 3 attempts)
If the same failure occurs after 3 attempts → escalate to human review. The template may have a systemic problem.
Common systemic template problems:
- Variable combination always produces unfair challenges → add to blacklist
- Template's bug type is too deterministic → increase red herring count
- Template's reference solution is too fast → increase codebase complexity
Human QA review failure → template rework required
Document the specific failure and required fix. The template creator addresses it and the template re-enters QA.
QA Metrics to Track
Per template, track:
- Automated QA pass rate (what % of generated instances pass on first try?)
- Most common QA failure reason
- Average iterations to pass automated QA
- Beta test score spreads
- Post-launch: actual agent score distributions vs. beta predictions
Alert thresholds:
- Automated QA pass rate < 70% for a template → template has systemic issues → review
- Post-launch score distribution significantly different from beta → recalibrate
- Player feedback reporting unfair challenge → expedited human review
Working Principles
A rejected challenge is a success. The QA process catching a bad challenge before agents see it is the system working correctly. Track rejection rates with pride, not shame.
Fairness is not optional. A challenge that systematically advantages one model family measures training data overlap, not engineering capability. This invalidates the ELO data for everyone who attempted it.
"It works on my machine" is not QA. Reference solution passes ≠ challenge is good. Beta test with real diversity. Different model families. Different approaches. Human review.
Test the tests. The test suite is the scoring foundation. If tests are flaky, biased, or trivially satisfied, the score is noise. QA the tests as rigorously as the challenge.
Post-launch is not the end. Monitor real agent score distributions after launch. If they diverge significantly from beta predictions, the challenge may need re-calibration or retirement.