Complete Test Code

Complete Test Code

Reliability is not achieved by careful coding alone — it is earned by testing. This skill packages the testing methodology that makes SQLite one of the most reliable pieces of software on Earth, and translates it into practices you can apply to any project, in any language, at a cost proportional to what's at stake.

Two jobs:

• Writing new tests with the rigor the code actually deserves.
• Auditing / hardening an existing suite — finding the gaps that let bugs through.

The goal is never "more tests." It is the right tests, where they matter, that would actually notice if the code were wrong.

Core philosophy — adopt this mental model first

These seven ideas are the whole skill in compressed form. Everything else is mechanics.

1. Test investment scales to blast radius, not to code size. What determines how much rigor a component deserves is the cost of being wrong — how widely it's deployed, how hard a bad release is to recall, and whether it persists state that "remembers" mistakes. A near-zero test ratio on a load-bearing module is the real red flag; an imperfect coverage number on a throwaway script is not.

2. The unhappy path is where reliability lives. Correct behavior on good input on a healthy machine is the easy part. The hard, valuable work is sane behavior under bad input and system malfunction (OOM, I/O errors, crashes, timeouts, corruption). Error-handling and cleanup code is the least-exercised and most dangerous code in any program, because it only runs when something already went wrong. Deliberately injecting failures and feeding garbage is the single highest-leverage shift most suites can make.

3. Independent, redundant verification beats one bigger suite. A single suite encodes its authors' assumptions, so its blind spots are correlated — bugs those assumptions hide stay hidden no matter how large it grows. Diversity of method (unit + contract + property + differential + fuzz) and of authorship (tests written from the spec, by someone other than the implementer) produce uncorrelated failure modes. When two independent checks disagree, that disagreement is a high-signal bug.

4. An oracle you didn't hand-write lets correctness testing scale. The hardest bug class is the wrong answer — crash-free, leak-free, yet incorrect. You can't hand-author expected results at scale, so generate them: a reference implementation, a prior version, your own unoptimized slow path, an inverse function (round-trip), or a metamorphic relation (a transform that must not change the result). Each turns millions of generated inputs into checkable correctness tests for free.

5. Coverage measures the test suite, not the product. Running under a coverage tool "is not a test of the code, it is a test of the test — a meta-test." Coverage proves code ran, never that a test would notice if it were wrong. Two disciplines give it meaning: dense assertions turn silent corruption into loud, located failures; mutation testing breaks the code on purpose to confirm a test fails. Prefer branch coverage over statement coverage, and validate the as-shipped build separately.

6. Make the safety net cheap, automatic, and tiered — or it won't run. The bottleneck is whether checks actually execute. Run a fast subset on every commit (minutes) and the full expensive suite (fuzz/soak/anomaly/matrix) before release. Bake leak detection into every run with zero setup. Distill billion-case fuzz campaigns into a few-thousand-case corpus that replays on every make test. On-by-default and fast-enough-to-be-habitual is what makes a technique real.

7. Automation catches anticipated failures; keep a human asking "is this really right?" Every assertion encodes a failure someone foresaw. SQLite deliberately does not automate its ~200-item release checklist, because a human reviewing output at the top level catches anomalies no assertion was written for. The same skepticism applies to a green CI check or AI-generated "passing" code.

START HERE — right-size the effort before writing anything

The fastest way to misuse this skill is to apply SQLite-grade rigor to a CRUD admin page, or Essential-only rigor to a payments ledger. Score the component (not the whole repo) on four axes, 1–5 each, and sum:

Two overrides bump a component up regardless of the sum:

• It's an attack surface for untrusted input → fuzzing + malformed-input tests become mandatory.
• It persists or replicates state that can't easily be recalled → integrity / atomicity / crash testing become mandatory.

Score per module, not per repo. A project can have an Extreme-tier storage layer and an Essential-tier admin UI. When unsure between two tiers, the cheap Essential items are almost never wrong; the expensive Extreme ones are wrong far more often than right for typical apps.

Full rubric, the "does it remember its mistakes?" heuristic, and worked examples (throwaway script → internal service → public library → migration layer → auth/billing): references/decision-framework.md.

Workflow A — writing new tests

1. Score the component (rubric above) → pick the tier. Note the two overrides.
2. Cover the unhappy path first. For each external dependency (allocation, I/O, network, DB, sub-service), write a test that makes it fail and assert a clean typed error plus consistent state after — not just the return code. This is where the bugs are.
3. Pin the boundaries. For every limit (documented or implicit), test max-allowed (succeeds), first-over (rejected), and the n−1 / n / n+1 neighbors. See references/fuzzing-and-malformed-input.md.
4. Build a generated oracle for any wrong-answer-sensitive logic (round-trip, reference impl, slow path, metamorphic relation) so you're not limited to hand-written cases. See references/oracles-and-differential.md.
5. Assert invariants in the code, not only in tests — preconditions, postconditions, "can't happen" guards. Run tests with assertions enabled. See references/dynamic-analysis.md.
6. Parameterize / table-drive so a few definitions yield many cases. Keep tests co-located and the suite fast.
7. Add the tier's Advanced/Extreme techniques (fault-injection loops, fuzzing, mutation, crash simulation) by following the matching reference file.
8. Verify the test would fail. Before trusting a new test, confirm it goes red when the behavior is broken (red-green). A test that can't fail protects nothing.

Workflow B — auditing / hardening an existing suite

1. Score each major component to know what rigor it should have, then measure the gap.
2. Run the audit checklist against the repo: copy assets/test-rigor-checklist.md in and answer each yes/no question. Each item names its tier, so you only owe the items at or below the component's tier.
3. Look for the classic holes, in priority order:
   • Bugs closed with no regression test (grep the tracker against the test diff history).
   • Failure paths never exercised — search for catch/except/rescue/error returns with no test that triggers them. (Coverage tools usually show these as the uncovered lines.)
   • Coverage reported as line not branch; or measured on a debug build that isn't shipped.
   • Tests that run code but assert nothing meaningful (mutation testing exposes these fast).
   • Leaks on error paths; missing teardown checks for handles/connections/threads/listeners.
   • One monolithic suite from one author/method (correlated blind spots).
4. Prove the gap with mutation testing on a critical module — surviving mutants are a ranked to-do list of missing assertions. See references/coverage-and-mutation.md.
5. Report findings by tier and blast radius, recommend the highest-yield fixes first, and (per philosophy #6) wire each fix into the default run so it can't be skipped.

The three tiers at a glance

A compact index; the full what/why/how-to-start for each technique lives in the theme references below. (E)=Essential (A)=Advanced (X)=Extreme.

• Regression & process: every bug → a permanent failing-then-passing test (E) · fast/full tiered runs (E) · CI matrix across OS/runtime/flags (E) · written release checklist + human review (E) · warning-clean multi-compiler builds (A). → references/regression-and-process.md
• Fault injection & resilience: inject failure at a seam and sweep it across every step (A) · transient vs persistent modes (A) · verify integrity + atomicity after failure (E) · always-on leak detection (A) · compound failures (A) · crash/power-loss simulation (X) · protocol-monitoring shim (X). → references/fault-injection-and-resilience.md
• Fuzzing & malformed input: boundary values both sides (E) · coverage-guided fuzzing on untrusted input under sanitizers (A) · reproducer corpus replayed every run (A) · malformed-artifact tests (A) · structure-aware / multi-surface fuzzing (X) · continuous fuzzing (X). → references/fuzzing-and-malformed-input.md
• Oracles & differential: differential vs reference/prior/slow-path (A) · optimization on-vs-off identical-output diff (A) · metamorphic & round-trip checks (A) · separate correctness from effectiveness tests (A). → references/oracles-and-differential.md
• Coverage & mutation: branch over statement coverage (E) · per-component targets (E) · keep & mark defensive code, don't delete for the number (A) · mutation testing (A) · MC/DC on critical predicates (X) · coverage as meta-test, validate shipped build (X). → references/coverage-and-mutation.md
• Dynamic analysis: assertions as executable contracts (E) · memory/UB/race analyzers on a subset pre-release (A) · debug allocator layering (A) · lock-ownership asserts (A) · config-matrix for impl-defined behavior (X). → references/dynamic-analysis.md
• Ecosystem tooling: concrete commands/libraries per language (JS/TS, Python, Go, Rust, Java, C/C++) for every technique above. → references/ecosystem-pointers.md

Anti-patterns — how this goes wrong (read before cargo-culting)

The honest failure modes, several called out by the SQLite authors themselves:

• The 590:1 ratio does not generalize. SQLite itself says 100% MC/DC is "probably not cost effective for a typical application." What transfers is the budgeting heuristic (effort ∝ blast radius), never the magnitude. Don't quote the ratio as a target.
• Coverage is not correctness. 100% coverage with weak assertions catches almost nothing — the code ran, nothing checked the result. Never let a percentage become the goal; it pressures people to delete guards and write assertion-poor tests. Mutation testing is the antidote.
• Fuzzing and 100% MC/DC pull against each other. Defensive "can't-happen" guards create unreachable branches that wreck coverage — but deleting them to hit the number is exactly what lets a fuzzer reach an unexpected state. Keep defensive code on attack surfaces, mark unreachable branches so coverage stays honest, and treat normal-use robustness and attack robustness as separate goals with separate budgets.
• Differential testing only works where outputs are supposed to agree. Quarantine known divergences (dialect, locale, float formatting, undefined ordering) or you drown in false positives. Likewise, keep effectiveness tests (that count cache hits / queries / sorts) out of any "optimization off" run — they fail by design and erode trust in the suite.
• Opt-in protection is no protection. A leak detector you have to enable, or a fuzzer you run once, gets skipped and catches nothing. If a check isn't wired into the default run, assume it isn't protecting you.
• Happy-path checks give false confidence. The value is in checking cleanup after an injected failure, and in compound failures (a fault during recovery from a prior fault).
• Static analysis isn't universally high-yield. SQLite found few bugs with it — and introduced more bugs mechanically silencing warnings than the analyzer ever found. Use cheap compiler warnings and type checkers, but review a warning-"fix" as carefully as a feature change, and don't refactor subtle working logic to appease a false positive.
• More of the same kind of test doesn't add coverage. Tests written from the implementation by its author inherit its blind spots. Independence — a different author, the spec instead of the code, a different method/oracle — is what adds real protection.

Reference files (hub-and-spoke — open what the task needs)

When you apply a non-obvious technique, briefly tell the user why it fits this component's tier — the reasoning is the transferable part, and it keeps the team from later ripping out rigor they don't understand.