Code Verifier

/code-verifier

Audit the GENUINENESS of evidence — not just that a command ran.

Master TOC

• Core principle
• The Three-Layer Gate
• Fake-Run Patterns
  • A. Tests that pass without testing
  • B. Scripts that fabricate results
  • C. Pipelines that short-circuit
  • D. ML-research specific
• When to invoke
• Output format when fake pattern found
• Anti-rationalisation
• Companion

Core principle

"Ran a command" ≠ "verified the work." A test that always passes (because it asserts True) proves nothing. A script that prints mAP = 0.85 from a hardcoded literal proves nothing. A pipeline that swallows exceptions silently and returns a default proves nothing.

This skill audits the GENUINENESS of evidence — not just that a command ran.

The Three-Layer Gate

Before any completion claim involving code / tests / results, run all three layers in order:

1. Layer 1 — Did it really run? Defer to superpowers:verification-before-completion. Run the exact command in this turn; read full output; check exit code. Skip if already cleared in this turn.
2. Layer 2 — Is the run genuine? Audit the artifact for the FAKE-RUN patterns below. If any match, the run is NOT genuine and the claim is unsupported until you fix and re-run.
3. Layer 3 — Does the run exercise the claim? Even a real run can fail to exercise the claim — e.g. test imports the wrong module, runs with pytest -k other. Check that the artifact's scope matches the claim.

Fake-Run Patterns

A. Tests that pass without testing

Detection commands:

# Tautological asserts
grep -rE "assert\s+(True|1\s*==\s*1)" tests/ src/ 2>/dev/null

# Tests with zero assertions
grep -rL "assert\|expect\|self\.assert" tests/ 2>/dev/null

# Bare-except in tests
grep -rE "except\s*:\s*pass|except\s+Exception:\s*pass" tests/ 2>/dev/null

# Skipped tests
grep -rE "@pytest\.mark\.skip|pytest\.skip\(\)" tests/ 2>/dev/null

# Mocking the SUT
grep -rE "mock\.patch.*$(basename module_under_test)" tests/ 2>/dev/null

B. Scripts that fabricate results

Detection commands:

# Suspicious literal numbers near metric prints
grep -rE "(mAP|acc|score|loss|recall|precision)\s*[=:]\s*[0-9.]" outputs/ scripts/ 2>/dev/null | grep -vE "for|while|range|len|shape|return"

# Or-fallback to numeric literals
grep -rE "= [^#]*\(\)\s*or\s+[0-9.]+" src/ scripts/ 2>/dev/null

# np.random in eval scripts (excluding seeded)
grep -rE "np\.random|torch\.rand|random\." scripts/ 2>/dev/null | grep -viE "seed|generator"

C. Pipelines that short-circuit

Detection commands:

# TODO/stub patterns near returns
grep -rEB1 "return.*#.*TODO|return.*#.*FIXME|return.*#.*placeholder" src/ scripts/ 2>/dev/null

# Swallowed exceptions
grep -rE "except.*:\s*(pass|return\s+(None|default|\{\}|0))" src/ scripts/ 2>/dev/null

# Disabled blocks
grep -rE "if\s+(False|0):" src/ scripts/ 2>/dev/null

# time.sleep outside tests
grep -rE "time\.sleep" src/ scripts/ --exclude-dir=tests 2>/dev/null

D. ML-research specific

Detection commands:

# Training loop without backward
grep -rEB3 "opt(imizer)?\.step\(\)" src/ scripts/ --include='*.py' | grep -A3 "\.step" | grep -L "\.backward" 2>/dev/null

# Eval on train loader
grep -rE "eval\(.*train_loader|test_metric.*train_data" src/ scripts/ --include='*.py' 2>/dev/null

# Cherry-picked seeds — max() over seed reports
grep -rE "best_(of|seed)|max\(.*seeds" src/ scripts/ --include='*.py' 2>/dev/null

# Hardcoded plot data near plt calls
grep -rB3 -A1 "plt\.(plot|bar|scatter)" scripts/ --include='*.py' 2>/dev/null | grep -E "= \[[0-9.,\s]+\]"

# Val/test split overlap
python3 -c "
import json
for p in ['data/splits.json', 'data_split.json']:
    try:
        d = json.load(open(p))
        v = set(d.get('val', d.get('validation', [])))
        t = set(d.get('test', []))
        print(f'{p}: val∩test = {len(v & t)} of {len(v)} val, {len(t)} test')
    except: pass
"

When to invoke

Output format when fake pattern found

[FAKE-RUN DETECTED]

Pattern: <category>.<subcategory>
File: path/to/file.py:line
Evidence:
  <exact line(s) showing the pattern>

Why this is fake:
  <one sentence explanation>

Required fix:
  <concrete change>

Re-run command after fix:
  <exact command>

Re-running NOT optional — the previous claim of <X> is NOT supported.

Anti-rationalisation

Companion

• superpowers:verification-before-completion — Layer 1 (real-run discipline)
• research-critic — audits inferential chain ON TOP of authentic artifacts
• always-on-verification rule (in this lib) — when to invoke

Provenance

Originally authored as a user-level always-on gate; consolidated into agent-harness for cross-project reuse. Pairs with research-critic for full claim-defensibility coverage.