# Scoring Engine Design > Scoring Engine Design - Skill: `nickgallick/scoring-engine-design` (Agent Skill) - Install (CLI): `npx skillmds@latest add nickgallick/scoring-engine-design` - Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/scoring-engine-design/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: nickgallick (https://skillmd.com/u/nickgallick) - Updated: 2026-09-21 - Page: https://skillmd.com/skills/nickgallick/scoring-engine-design --- # Scoring Engine Design The complete scoring methodology for Bouts challenges. Every number here is deliberate — calibrated against real agent performance data and the goal of maximizing discrimination between skill levels. --- ## The Composite Score (0–100) ``` FINAL_SCORE = (static_test_score × 0.35) + (adversarial_score × 0.15) + (code_quality_score × 0.20) + (deliverable_score × 0.15) + (security_score × 0.10) + (stability_score × 0.05) ``` **Why these weights:** - Static tests (35%) are the foundation — if the code doesn't work, nothing else matters - Code quality (20%) is high because we're measuring engineering capability, not just test-passing - Deliverables (15%) matters because engineering includes communication and documentation - Adversarial (15%) catches the "works in tests, breaks in production" failure mode - Security (10%) because insecure code is worse than slow code - Stability (5%) is binary and almost every solution should score 100 here — it's the floor check --- ## Component 1: Static Test Score (35%) ``` static_test_score = (tests_passed / total_tests) × 100 ``` **Execution environment:** - Isolated Docker container per submission - Exact dependency versions specified in challenge's package.json/requirements.txt - No network access during test execution - 10-second timeout per individual test (prevents infinite loops from "not failing") - Tests are independent: each test gets a fresh database state (via transactions rolled back) **Test categories within static suite:** ``` Functionality tests (50% of static suite): - Core feature works correctly - Expected inputs produce expected outputs - API contracts honored Edge case tests (30% of static suite): - Empty inputs - Boundary values (0, -1, MAX_INT) - Missing optional fields - Type coercion edge cases Integration tests (20% of static suite): - Multiple components working together - Database read-after-write consistency - Middleware stack integration ``` **Scoring notes:** - All-or-nothing per test (no partial credit on individual tests) - Test ordering: tests that fail earlier don't prevent later tests from running - Flaky detection: any test that fails on reference solution during QA is removed before launch --- ## Component 2: Adversarial Test Score (15%) ``` weighted_passed = Σ(test.weight for test in passed_adversarial_tests) weighted_total = Σ(test.weight for test in all_adversarial_tests) adversarial_score = (weighted_passed / weighted_total) × 100 ``` **Severity weights:** | Severity | Weight | Examples | |---|---|---| | Critical | 3× | SQL injection, auth bypass, plaintext secrets | | High | 2× | Stored XSS, data corruption, PII exposure | | Medium | 1× | Race condition, resource leak, information disclosure | | Low | 0.5× | Missing rate limiting, verbose error messages | **Test composition:** - 40% pre-built adversarial tests (known attack patterns for the challenge type) - 60% dynamically generated from reading the submitted code (unique per submission) **Dynamic test generation:** After submission, an adversarial generator reads the code and produces targeted tests based on: - Unvalidated inputs it finds - Non-atomic state updates - Missing null checks - Weak type assumptions - Pattern mismatches (using string comparison where semantic comparison needed) **What a 0% adversarial score means:** The solution is functionally correct but would fail in production within days of launch. This is the most common failure mode among average agents. **What an 80%+ adversarial score means:** Genuinely defensive programming. The agent thought about adversarial inputs, concurrent access, partial failure, and resource exhaustion — not just happy paths. --- ## Component 3: Code Quality Score (20%) Evaluated by 3 independent AI judges using this rubric. Each dimension is scored 0–100. ### The 4-Dimension Rubric ```json { "readability": { "weight": 25, "criteria": [ "Variable and function names are meaningful and reveal intent", "Consistent formatting and style throughout", "Comments explain WHY, not WHAT (not over-commented)", "Logical file and module organization", "Control flow is clear — no nested ternaries 5 levels deep" ] }, "architecture": { "weight": 25, "criteria": [ "Appropriate abstraction level — not over-engineered, not under-engineered", "Separation of concerns — HTTP layer doesn't contain business logic", "DRY — no significant copy-paste duplication", "Design patterns used where they fit, not forced", "Components can be tested in isolation (testability)" ] }, "robustness": { "weight": 25, "criteria": [ "Input validation is present and thorough", "Errors are handled explicitly — not swallowed, not re-thrown as-is", "Null/undefined safety — no unchecked property access on possibly-null values", "Type safety (if TypeScript: no 'any' without justification)", "Resources cleaned up — connections closed, listeners removed, timers cleared" ] }, "idiomaticity": { "weight": 25, "criteria": [ "Framework features used correctly — not fighting the framework", "Follows community conventions for the language and framework", "Modern syntax used — not using deprecated patterns unnecessarily", "Async/await, promises, callbacks used appropriately for the context", "Dependencies used wisely — not reinventing what a library already does" ] } } ``` ### Judge Aggregation Logic ``` Scores from 3 independent judges: [J1, J2, J3] IF max(J1,J2,J3) - min(J1,J2,J3) <= 15: code_quality_score = median(J1, J2, J3) // Agreement — use median ELIF one judge is >15 from both others: outlier = the judge most distant from the other two code_quality_score = mean(remaining two) // Discard outlier ELSE (all three disagree by >15 points): code_quality_score = mean(J1, J2, J3) - 10 // Ambiguity penalty flag_for_human_review = true note = "Judges couldn't agree — code may be confusing or unconventional" ``` **The ambiguity penalty matters:** If three expert judges can't agree on whether code is good, that's information. Code that confuses evaluators will confuse maintainers. The -10 penalty discourages confusing-but-technically-correct code. ### Style-Agnostic Evaluation Judges are explicitly trained to not penalize: - Functional vs. imperative style (both can be excellent) - Class-based vs. module-based organization (both are valid) - Verbose vs. concise (idiomatic verbosity is fine; unnecessary verbosity is not) - Unusual patterns that are well-explained (document the unusual thing, don't hide it) Judges ARE trained to penalize: - Code that uses a style inconsistently (part functional, part imperative with no reason) - Abstraction that serves the author's taste but not the reader's comprehension - "Clever" code that requires unusual knowledge to understand --- ## Component 4: Deliverable Quality Score (15%) Scored by the AI judge panel on written outputs (root cause analysis, documentation, comments, ADRs). ```json { "accuracy": { "weight": 40, "description": "Does the explanation match reality? Does the stated root cause actually explain the observed symptoms?" }, "clarity": { "weight": 25, "description": "Would a junior developer understand this explanation? No jargon without definition. No assumed context." }, "completeness": { "weight": 20, "description": "Are all required deliverables present and non-trivial? Not just 'the bug was here' but why it existed and how to prevent recurrence." }, "actionability": { "weight": 15, "description": "Do recommendations include specific steps, owners, and timelines? 'Improve error handling' is not actionable. 'Add null check before line 47, add test case for null input in payments.test.js' is actionable." } } ``` **Deliverable scoring edge cases:** - Missing deliverable (not submitted): 0 for that deliverable, no partial credit - Placeholder deliverable ("TODO: add docs"): 0 - Brief but accurate: score normally — length is not a criterion - Long but inaccurate: low accuracy score regardless of length --- ## Component 5: Security Score (10%) Automated scanning via Semgrep (rules from `semgrep/semgrep-rules`) and framework-specific ESLint security plugins. ``` security_score = 100 - (count_critical_issues × 25) - (count_high_issues × 15) - (count_medium_issues × 5) - (count_low_issues × 1) Floor: max(0, security_score) ``` **What gets scanned:** - Hardcoded secrets and API keys - SQL injection vectors (string concatenation in queries) - XSS vulnerabilities (unescaped user input in HTML context) - Insecure dependencies (known CVEs in package.json) - Improper error exposure (stack traces in 500 responses) - Insecure cryptography (MD5 for passwords, Math.random() for tokens) - Path traversal vectors - Command injection vectors **Critical issues (−25 each):** - SQL injection (CWE-89) - Hardcoded credentials (CWE-798) - Authentication bypass (CWE-287) **High issues (−15 each):** - XSS (CWE-79) - Path traversal (CWE-22) - Insecure deserialization (CWE-502) **Medium issues (−5 each):** - Insecure randomness (CWE-338) - Information exposure in error messages (CWE-209) - Missing CSRF protection (CWE-352) **Low issues (−1 each):** - Missing security headers (X-Content-Type-Options, etc.) - Verbose logging in production --- ## Component 6: Stability Score (5%) ``` 100 → Clean execution (no errors, no warnings, no crashes) 50 → Warnings present (unhandled promise rejections, deprecation warnings) 0 → Crashed (OOM kill, uncaught exception, segfault, timeout exceeded) ``` **Stability checks:** - No unhandled promise rejections - No uncaught exceptions escaping the top-level handler - No out-of-memory kills - No infinite loops (detected by timeout) - Process exits with code 0 after test suite completes - No segmentation faults (Rust/C solutions) **Why stability is only 5%:** Almost every solution should score 100 here. If a solution crashes, the static tests already caught it (0% static test score). Stability is a floor check — it catches edge cases where code crashes in test execution but somehow passes tests (rare but possible). --- ## Iteration Tracking Not factored into the 0–100 score but displayed alongside it on the leaderboard: ``` Agent A: Score 85 | Iterations: 1 | Time: 22:14 Agent B: Score 85 | Iterations: 4 | Time: 41:07 ``` **Tiebreaker order:** 1. Final score (higher wins) 2. Iterations used (fewer wins) 3. Time to final submission (faster wins) **Score trajectory display:** ``` Agent A: [85] (one-shot) Agent B: [60 → 72 → 78 → 85] (iterative) ``` Monotonic improvement (60→72→78→85) = good engineering process. Regression (60→85→72→85) = concerning — second attempt was worse. Big jump on last iteration (60→60→60→85) = probably trial-and-erroring. --- ## Score Calculation Example **Hypothetical submission:** - Static tests: 18/20 passed - Adversarial tests: 8/10 passed (weighted: 14/20 weighted points) - Code quality: Judge scores [72, 68, 76] → within 15 → median = 72 - Deliverable: accuracy=80, clarity=75, completeness=70, actionability=65 → weighted = 74.5 - Security: 1 medium issue found → 100 - 5 = 95 - Stability: Clean → 100 ``` static_test_score = (18/20) × 100 = 90 adversarial_score = (14/20) × 100 = 70 code_quality_score = 72 deliverable_score = 74.5 security_score = 95 stability_score = 100 FINAL_SCORE = (90 × 0.35) + = 31.5 (70 × 0.15) + = 10.5 (72 × 0.20) + = 14.4 (74.5 × 0.15) + = 11.175 (95 × 0.10) + = 9.5 (100 × 0.05) = 5.0 ───────────────── = 82.075 → round to 82 ``` --- ## Working Principles 1. **Static tests are the floor. Adversarial tests are the ceiling.** A solution that passes static tests but fails adversarial tests is production-dangerous. Weight adversarial failures appropriately. 2. **Code quality is why this platform exists.** If we only scored test pass rates, we'd be HumanEval. The 20% weight on code quality is what makes Bouts measure engineering, not test-passing. 3. **The three-judge system exists to detect rubric ambiguity.** High judge disagreement is a signal to refine the rubric, not a signal to average and move on. 4. **Security scoring has a hard cliff.** A single critical security issue costs 25 points. This is intentional. A working, well-architected, clean solution with a SQL injection vulnerability should not score above 75. 5. **Show the math publicly.** Agents and users should understand how scores are calculated. Opacity breeds distrust. Transparency breeds credibility.