bobmatnyc Mutation Testing
A second-order test-quality signal. Line coverage tells you a line executed; mutation testing tells you whether any assertion would fail if that line were wrong. Use it to audit the strength of an existing suite — most valuable on logic-dense code, applied on-demand, never as a default blocking CI gate.
What it does
A mutation tool makes small automatic edits ("mutants") to your source, e.g.:
<-><=(boundary / off-by-one)and->or(predicate logic)==->!=(negated condition)return x->return None,+->-, deleting a statement- removing a list/dict entry (allowlist/denylist coverage)
For each mutant it re-runs the test suite:
- Killed — at least one test fails. Good: your tests detect that change.
- Survived — all tests still pass. Bad: the mutant models a real bug your suite would not catch.
Kill rate = killed / total mutants. It measures test effectiveness, not code execution. You can have 100% line coverage with a 40% kill rate: every line runs, but half the logic changes go undetected because the tests assert too little.
When to use it (and when not to)
Use it to: audit a critical module before relying on it (payment math, auth predicates, allowlists gating filesystem/network access); harden tests after a near-miss bug; turn "we have 90% coverage" into "here are the specific behaviors no test pins down".
Do NOT use it as: a default blocking CI gate (runtime is N mutants x full suite); a
metric to chase to 100% (some mutants are equivalent — semantically identical — and can
never be killed).
Workflow
- Choose one logic-dense module with existing green tests.
- Run a mutation tool scoped to that module (Python:
mutmutorcosmic-ray; JS/TS:stryker). - List SURVIVED mutants. Each names a behavior no test verifies.
- For each survivor that represents a real bug, add or strengthen an assertion to kill it.
- Re-run; confirm the kill rate rose. Stop when survivors are only equivalent mutants.
A rising kill rate on a critical module is the goal — not a perfect score.