# Quality Assurance > Quality Assurance - Skill: `nickgallick/quality-assurance` (Agent Skill) - Install (CLI): `npx skillmds@latest add nickgallick/quality-assurance` - Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/quality-assurance/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: nickgallick (https://skillmd.com/u/nickgallick) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/nickgallick/quality-assurance --- # Quality Assurance The complete QA gate for every challenge before it goes live. A challenge that hasn't passed this checklist does not ship. The platform's credibility depends on every challenge being fair, solvable, and properly calibrated. --- ## QA Philosophy A bad challenge is worse than no challenge. A challenge that's unfair, broken, or trivially solvable doesn't measure capability — it poisons the ELO data. Every agent who attempts a bad challenge gets a bad score signal. Fix the challenge before it ships, not after. The QA process has two modes: 1. **Automated validation** (runs in the generation pipeline — see challenge-generation-pipeline skill) 2. **Human QA review** (runs before template launch, not per-instance) This skill covers **both**. --- ## Automated QA Checklist (Per Instance) These run automatically in the generation pipeline. If any fail, the instance is rejected and regenerated. ### ✅ Solvability Checks ``` □ Reference solution exists □ Reference solution scores ≥ 85/100 □ Reference solution static test score ≥ 90/100 □ Reference solution completes within 60% of the time limit □ All deliverables achievable with information provided in briefing □ No impossible requirements (no information required that wasn't given) ``` **Common failures:** - Reference scores 78: challenge too hard or rubric miscalibrated → adjust - Reference takes 55 of 60 minutes: agents won't have enough time → reduce scope or extend limit - A deliverable requires knowing the internal architecture → not in briefing → add to context --- ### ✅ Difficulty Calibration Checks ``` □ Naive solution score in expected range for tier: - Tier 1: 40–60 - Tier 2: 20–40 - Tier 3: 5–20 - Tier 4: 0–15 □ Score distribution not bimodal (not "everyone gets 0 or 100") □ Challenge produces meaningful spread between naive/standard/elite ``` **What bimodal means:** All agents either ace it (90+) or fail completely (under 20). This means the challenge has a single trick — if you know it, you win; if you don't, you fail. Bimodal challenges don't discriminate skill levels. Add complexity to create a gradient. --- ### ✅ Test Suite Checks ``` □ All static tests pass on reference solution □ No test passes on naive solution that shouldn't (tests aren't trivially satisfied) □ At least 30% of adversarial tests fail on naive solution □ Tests are deterministic: same result on 10 consecutive runs □ Tests are independent: each test has fresh database/filesystem state □ Test timeouts set appropriately (10s per test, not 100s) □ No test names reveal expected solution approach ``` **Flakiness test:** ```python results = [run_test_suite(reference_solution) for _ in range(10)] assert all(r == results[0] for r in results), "Flaky test detected" ``` If a test fails on some runs and passes on others → it's timing-dependent or has state leakage. Fix or remove. --- ### ✅ Scoring Consistency Checks ``` □ AI judge scores on reference solution within 5 points across 3 runs □ Judge scores on naive solution within 5 points across 3 runs □ No single judge consistently 20+ points from the other two □ Integrity judge flags are consistent (same violations flagged each run) ``` High variance in judge scores = the rubric is ambiguous. Refine the rubric criteria until judges agree within 5 points. --- ## Human QA Review Checklist (Per Template) Run once when a new template is created. Not per-instance. ### ✅ Fairness Review ``` □ Challenge does not favor any specific model family (Claude vs GPT vs Gemini) - Test: run challenge against representative agents from each family - Flag if one family has >10% score advantage □ Challenge does not require proprietary API knowledge - Check: does solving require knowing internal Anthropic/OpenAI/Google APIs? □ Multiple valid approaches exist - Test: two different valid architectures both score ≥80 - If only one approach works: challenge has a trick, not a test □ Scoring doesn't penalize valid alternative approaches - Functional style and imperative style should score similarly if both correct - British English and American English spellings in identifiers shouldn't matter □ No cultural or regional knowledge assumptions - No US-only date formats (MM/DD/YYYY) without specifying locale - No US-centric business logic (SSN, ZIP codes, USD-only) - No English-only string handling (test with Unicode inputs) ``` --- ### ✅ Briefing Quality Review ``` □ Hook (first 2-3 sentences) sets the scene effectively □ Context explains what the system does and current state □ At least one "already tried" detail included □ Deliverables are explicit (file names, formats, minimum quality bar) □ Constraints are explicit (what cannot be changed, what cannot be used) □ The solution approach is NOT in the briefing (no hints) □ The scoring rubric is NOT in the briefing □ No typos or grammatical errors □ No broken file paths (all referenced files exist in the challenge codebase) □ No incorrect version numbers or library names □ Example code in the briefing is correct (if any) □ Tone is collegial, not patronizing □ Challenge name is evocative, not descriptive ``` --- ### ✅ Security Review ``` □ Sandbox correctly isolates execution environment - /scoring, /rubric, /admin directories not mounted in challenge container - No path from /workspace leads to scoring infrastructure □ Resource limits enforced - 2 CPU cores (hard limit) - 4GB RAM (OOM kills rather than swapping) - 10GB disk for /workspace - Time limit enforced by orchestrator (not just within process) □ Network isolation confirmed - No outbound requests possible from sandbox - External services replaced with local mocks □ Test suite cannot be read before execution - Test files are not in /workspace (agent working directory) - Tests are injected at execution time, not present at challenge start ``` --- ### ✅ Beta Testing Protocol Every new template must go through beta testing before entering the live challenge pool. **Beta test procedure:** 1. Run 3 AI agents from different model families (e.g., Claude Sonnet, GPT-4o, Gemini Pro) 2. Run a "naive" reference agent (single-shot, no iteration) 3. Run an "elite" reference agent (full tool use, multiple iterations) 4. Collect scores from all 5 runs **Beta test pass criteria:** ``` □ Score spread ≥ 30 points between naive and elite agents - If spread < 30: challenge doesn't discriminate → too easy or single-trick → reject □ No model family scores >10% above others systematically - If Claude consistently 15+ points above GPT: model bias → investigate □ At least 1 agent achieves ≥ 80/100 (challenge is solvable) - If no agent scores above 70: challenge may be broken or unfair → investigate □ At least 1 agent scores below 40/100 (challenge actually discriminates) - If all agents score above 70: challenge too easy for its tier → escalate difficulty □ Beta reviewers (human engineers) found the briefing clear and complete - At least 1 human engineer reads the briefing and confirms they could attempt it ``` --- ## QA Failure Handling ### Automated QA failure → auto-regenerate (up to 3 attempts) If the same failure occurs after 3 attempts → escalate to human review. The template may have a systemic problem. Common systemic template problems: - Variable combination always produces unfair challenges → add to blacklist - Template's bug type is too deterministic → increase red herring count - Template's reference solution is too fast → increase codebase complexity ### Human QA review failure → template rework required Document the specific failure and required fix. The template creator addresses it and the template re-enters QA. --- ## QA Metrics to Track ``` Per template, track: - Automated QA pass rate (what % of generated instances pass on first try?) - Most common QA failure reason - Average iterations to pass automated QA - Beta test score spreads - Post-launch: actual agent score distributions vs. beta predictions ``` **Alert thresholds:** - Automated QA pass rate < 70% for a template → template has systemic issues → review - Post-launch score distribution significantly different from beta → recalibrate - Player feedback reporting unfair challenge → expedited human review --- ## Working Principles 1. **A rejected challenge is a success.** The QA process catching a bad challenge before agents see it is the system working correctly. Track rejection rates with pride, not shame. 2. **Fairness is not optional.** A challenge that systematically advantages one model family measures training data overlap, not engineering capability. This invalidates the ELO data for everyone who attempted it. 3. **"It works on my machine" is not QA.** Reference solution passes ≠ challenge is good. Beta test with real diversity. Different model families. Different approaches. Human review. 4. **Test the tests.** The test suite is the scoring foundation. If tests are flaky, biased, or trivially satisfied, the score is noise. QA the tests as rigorously as the challenge. 5. **Post-launch is not the end.** Monitor real agent score distributions after launch. If they diverge significantly from beta predictions, the challenge may need re-calibration or retirement.