Old Coder

Old Coder: Evidence-First Development

An old coder's strategy for the agent era: don't read the code — make it run the gauntlet. This skill makes coding agents prove their work through executable specifications and evidence reports rather than relying on human code review.

When to Use

• User explicitly asks for high-assurance or evidence-first work ("reliable", "TDD", "prove it works", "I won't read the code")
• Changes touching high-stakes domains: money, auth, data loss, concurrency, public API
• When you need to produce an executable spec (SPEC) and evidence report (EVIDENCE) instead of relying on code review
• For routine changes where the user just wants normal tests, write good tests directly instead of invoking this loop

Anti-Rationalization Table

Workflow: The Evidence-First Loop

SPEC → (human approves spec, not code) → RED → GREEN → REFACTOR → GAUNTLET → EVIDENCE
                                          ↑_____________________|
                                              repeat per behavior

Phase 1: SPEC — The Only Thing the Human Reads Before Code

Goal: Turn the request into executable acceptance criteria before touching implementation.

Steps:

1. Write behaviors as Gherkin-style scenarios or a named test list — concrete inputs, concrete expected outputs, edge cases, and error cases.
2. Include what the change must NOT do (invariants: existing tests, public API signatures, performance budgets).
3. Include the setup plan: tools to install, git usage, files the gauntlet will add, and every new dependency with a one-line justification.
4. Write the spec to a file at an absolute path (so it's clickable in terminal).
5. Show the spec to the human in plain language and get explicit approval before writing implementation.
6. In autonomous mode: state the spec and proceed, but EVIDENCE must record spec approval: not obtained (autonomous run) with lower confidence.

Anti-gaming: An answer to a question is not an approval. Questions and approval are two exchanges — fold answers in, show revised spec, ask again.

Phase 2: RED — Prove Each Test Can Fail

Goal: Write the test for one behavior, run it, and watch it fail before writing implementation.

Steps:

1. If module doesn't exist, create a stub that raises (e.g., NotImplementedError) so test fails on behavior, not import.
2. Run the new test individually and observe failure.
3. If a new test passes immediately: it's either vacuous or behavior already exists. Prove it — break implementation with a throwaway mutant, watch test fail, restore.
4. Record pre-existing behavior kept as regression armor.

Phase 3: GREEN — Minimal Implementation

Goal: Write the least code that makes the failing test pass.

Steps:

1. Implement minimal code for the current behavior.
2. Run the full suite, not just the new test.
3. All tests must pass.

Phase 4: REFACTOR — Clean Up Under Green

Goal: Improve names, extract duplication, simplify structure while suite is green.

Rules:

• Implementation refactors touch no test files.
• Test-structure refactors (helpers, fixtures) allowed as separate step: assertions unchanged, suite green before and after, then rerun mutation to confirm tests still kill.
• Anything requiring editing an assertion is a behavior change — goes back to SPEC.
• Run suite after each refactor.

Phase 5: GAUNTLET — The Constraint Stack

Goal: Run every applicable layer after all spec behaviors are green. Scale to task (Tier 1/2/3), never skip silently.

Calibration:

• Tier 1 (trivial): Full suite + lint. No new tests required, state why.
• Tier 2 (normal): Full loop. Bug fixes MUST start with RED test reproducing the bug.
• Tier 3 (high stakes — money, auth, data loss, concurrency, public API): Full loop + property-based tests + mutation testing (tool-based) + adversarial pass — explicitly try to break your own implementation with hostile inputs before declaring done. Write a short failure model listing ways this change can hurt, and for each mode add a layer that catches it.

Phase 6: EVIDENCE — The Only Thing the Human Reads After Code

Goal: End with a report the human can trust without opening a single source file.

Required Sections:

1. Spec approval status: Obtained from user / not obtained (autonomous) with confidence downgrade
2. Source state: Commit SHA or sha256 tree hash (persist computation as script)
3. Toolchain: Pinned versions file (e.g., requirements-dev.txt)
4. Entry point: Single command that reruns every layer (e.g., tools/gauntlet.sh)
5. Independent verification: not performed / passed / failed / blocked (Tier 3 protocol in verifier.md)

Spec → Test Mapping Table:

Gauntlet Results Table (all from ONE final fresh run):

Skipped layers: List each with reason (or "none").

Honest notes: Failures hit during task and how resolved, spec revisions, anything reducing confidence.

Code Examples

Python Project Setup (pyproject.toml)

[project]
name = "my-project"
version = "0.1.0"
dependencies = []

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "pytest-cov>=5.0",
    "pytest-randomly>=3.15",
    "hypothesis>=6.100",
    "mutmut>=3.0",
    "mypy>=1.10",
    "ruff>=0.6",
    "pip-audit>=2.7",
    "gitleaks>=8.0",
]

[tool.pytest.ini_options]
addopts = "-q --randomly-seed=last"
testpaths = ["tests"]
python_files = ["test_*.py"]
python_functions = ["test_*"]

[tool.mutmut]
source_paths = ["src/"]
tests_dir = "tests/"
runner = "python -m pytest"

[tool.coverage.run]
source = ["src"]
branch = true

[tool.coverage.report]
fail_under = 100
show_missing = true

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

[tool.ruff]
target-version = "py311"
line-length = 100
select = ["E", "F", "I", "UP", "B", "C4", "PTH", "T20", "ARG", "SIM", "RUF", "PERF"]
ignore = []

Gauntlet Entry Point Script (tools/gauntlet.sh)

#!/usr/bin/env bash
set -euo pipefail

# Freshness by mechanism: delete stale artifacts from previous runs
rm -rf .coverage coverage.xml .mutmut-cache htmlcov .pytest_cache

echo "=== GAUNTLET START ==="
echo "Source state: $(git rev-parse HEAD 2>/dev/null || sha256sum $(find . -type f -name '*.py' | sort) | sha256sum | cut -d' ' -f1)"

# 1. Full test suite
echo "--- Tests ---"
python -m pytest -q

# 2. Static types
echo "--- Types ---"
python -m mypy src/

# 3. Lint + format
echo "--- Lint ---"
python -m ruff check .
python -m ruff format --check .

# 4. Changed-line coverage (requires baseline)
echo "--- Changed-line Coverage ---"
python -m pytest --cov=src --cov-branch --cov-report=term-missing --cov-fail-under=100

# 5. Mutation testing
echo "--- Mutation ---"
python -m mutmut run --paths-to-mutate=src/

# 6. Property-based tests (included in pytest run above)
echo "--- Properties ---"
python -m pytest -q -k "property"

# 7. Real execution
echo "--- Real Execution ---"
python -m src.cli --help  # or run actual CLI/app on realistic input

# 8. Supply chain & secrets
echo "--- Supply Chain ---"
python -m pip_audit
gitleaks detect --source . --verbose --redact

# 9. Suite health
echo "--- Suite Health ---"
python -m pytest -q --randomly-seed=last --randomly-dont-reorganize

echo "=== GAUNTLET COMPLETE ==="

SPEC Template (spec.md)

# SPEC: <Task Name>

**Tier**: 1 | 2 | 3
**Approval**: [ ] Obtained from user / [ ] Not obtained (autonomous)

## Behaviors (Gherkin-style)

### Feature: <Capability in user language>

#### Scenario: <One concrete behavior>
**Given** <concrete starting state>
**When** <concrete action with concrete input>
**Then** <concrete observable outcome, exact values>

#### Scenario: <Error case>
**Given** <starting state>
**When** <invalid/hostile input>
**Then** <exact error type/message/status, and what state must NOT change>

## Negative Constraints (Must NOT)

| Invariant | Verification Method |
|-----------|---------------------|
| Existing tests still pass | Full suite baseline |
| Public API signatures unchanged | API compatibility check (griffe) |
| No new network/filesystem/env usage | Capability diff |

## Setup Plan

- Tools to install: (from `requirements-dev.txt`)
- Git: init / checkpoint commit cadence
- Files gauntlet will add: `tests/`, `tools/gauntlet.sh`, `tools/mutants.py`
- New dependencies: (each with one-line justification)

## Failure Model (Tier 3 only)

| Failure Mode | Layer That Catches It |
|--------------|----------------------|
| Race condition | `go test -race` / threading stress + rerun |
| Parser edge case | Property-based tests (hypothesis) |
| Silent production failure | Observability assertions in tests |
| Rollback failure | Migration rehearsal test |

EVIDENCE Template (evidence.md)

## Evidence Report — <Task Name> (Tier <1|2|3>)

- Spec approval: <obtained | not obtained (autonomous)>
- Source state: <commit SHA | tree hash>
- Toolchain: <requirements-dev.txt>
- Entry point: `./tools/gauntlet.sh`
- Independent verification: <not performed | passed | failed | blocked>

### Spec → Test Mapping

| Scenario | Test | Status |
|---|---|---|
| Happy path: divide(10, 2) = 5 | `test_math.py::test_divide_happy` | pass |
| Error: divide(1, 0) raises ZeroDivisionError | `test_math.py::test_divide_by_zero` | pass |
| Must NOT: change existing API | `griffe` diff | pass |

### Gauntlet (Final Fresh Run)

| Layer | Command | Result |
|---|---|---|
| Tests | `pytest -q` | 47 passed, 0 failed |
| Types | `mypy src` | 0 errors |
| Lint | `ruff check . && ruff format --check .` | 0 warnings |
| Changed-line coverage | `pytest --cov=src --cov-fail-under=100` | 31/31 lines, 20/20 branches |
| Mutation | `mutmut run` | 22/22 killed |
| Property-based | `pytest -k property` | 5 properties, 200 examples each |
| Real execution | `python -m src.cli 10 2` | `5.0` |
| Supply chain | `pip-audit && gitleaks detect` | 0 vulns; 0 new deps |
| Suite health | `pytest --randomly-seed=last` | seed 42, all passed |

### Skipped Layers
- none

### Honest Notes
- Mutation runner initially had cache bug (fixed in tools/mutants.py with mtime pinning)
- Spec revised once: added negative constraint for env var access

Manual Mutation Runner (tools/mutants.py)

#!/usr/bin/env python3
"""
Manual mutation runner for ecosystems without a mature mutation tool.
Persisted in repo so EVIDENCE is reproducible.
Proves it executed each mutant via mtime pinning + cache check.
"""
import subprocess
import sys
import os
import hashlib
from pathlib import Path

MUTANTS = [
    # (file, original_snippet, mutated_snippet, description)
    ("src/rate_limiter.py", "if current >= limit:", "if current > limit:", "off-by-one: >= -> >"),
    ("src/rate_limiter.py", "return tokens >= cost", "return tokens > cost", "off-by-one: >= -> >"),
    ("src/rate_limiter.py", "self.tokens = min(self.tokens + rate, capacity)", "self.tokens = min(self.tokens + rate, capacity - 1)", "capacity off-by-one"),
    ("src/rate_limiter.py", "async def acquire", "async def acquire_not_called", "delete method"),
    ("src/rate_limiter.py", "return True", "return False", "flip boolean return"),
]

def file_hash(path: Path) -> str:
    return hashlib.md5(path.read_bytes()).hexdigest()

def run_tests() -> bool:
    result = subprocess.run([sys.executable, "-m", "pytest", "-q"], capture_output=True)
    return result.returncode == 0

def main():
    src_files = [Path(m[0]) for m in MUTANTS]
    original_hashes = {f: file_hash(f) for f in src_files}

    killed = 0
    total = len(MUTANTS)

    for file_path, original, mutated, desc in MUTANTS:
        path = Path(file_path)
        content = path.read_text()

        if original not in content:
            print(f"FAIL: mutant pattern not found: {desc}")
            sys.exit(1)

        # Apply mutant
        mutated_content = content.replace(original, mutated, 1)
        path.write_text(mutated_content)

        # Prove execution: mtime must change, no bytecode cache reuse
        new_hash = file_hash(path)
        if new_hash == original_hashes[path]:
            print(f"FAIL: mutant did not change file hash: {desc}")
            sys.exit(1)

        print(f"Testing mutant: {desc}")
        if not run_tests():
            print(f"  KILLED")
            killed += 1
        else:
            print(f"  SURVIVED - missing test!")
            # Restore and fail
            path.write_text(content)
            print(f"Manual mutation: {killed}/{total} killed")
            sys.exit(1)

        # Restore
        path.write_text(content)
        restored_hash = file_hash(path)
        if restored_hash != original_hashes[path]:
            print(f"FAIL: restore mismatch for {desc}")
            sys.exit(1)

    print(f"Manual mutation: {killed}/{total} killed")
    # Final verification
    if not run_tests():
        print("FAIL: suite not green after restore")
        sys.exit(1)

if __name__ == "__main__":
    main()

Verification Checklist

Run this checklist before claiming the task is complete:

• SPEC written to absolute path and contains: behaviors (Gherkin/named tests), negative constraints, setup plan with justified dependencies, failure model (Tier 3)
• Human explicitly approved SPEC before any implementation (or autonomous mode noted in EVIDENCE)
• RED phase: Every new test observed failing individually before implementation
• GREEN phase: Minimal implementation, full suite passes
• REFACTOR phase: Only under green, assertions frozen, mutation rerun after test-structure changes
• GAUNTLET: Every applicable layer run, all exit non-zero on failure, no silent skips
• Changed-line coverage: Gates at 100% with --cov-fail-under (not just reports %)
• Mutation testing: Tool-based preferred; manual with persisted runner that proves execution
• EVIDENCE report: Single final fresh run, reproducible entry point, spec→test mapping, all layers with commands+results, skipped layers with reasons, honest notes
• Anti-gaming: No weakened tests, no simultaneous test+impl edits, no mocking unit under test, no coverage chasing, no unrun layers reported
• Tier 3: Failure model written, adversarial pass performed, independent verification protocol acknowledged
• Independent verification (Tier 3): verifier.md protocol executed or explicitly marked not performed with confidence downgrade

References

• Original repo: https://github.com/AmazingAng/old-coder
• Gauntlet tooling by ecosystem: references/gauntlet.md
• Verifier protocol (Tier 3): references/verifier.md
• Verifier case study: references/verifier-case-study.md
• Demo project: demo-rate-limiter/ (shows 41 tests, 100% coverage, 22/22 mutants killed)