Mutation Testing

Mutation Testing (Verbose)

Core Patterns

Why Coverage Is the Wrong Question

Line coverage answers "did this line execute during the test run". That is a much weaker claim than "if this line were wrong, a test would fail" — and the second is the only claim anyone actually cares about. The gap between them is where escaped bugs live.

Mutation testing closes the gap operationally. It edits the production code in small, syntactically valid ways — a mutant — reruns the tests, and asks whether anything went red. A mutant that causes a failure is killed. A mutant that leaves the suite green survived, and a survivor is direct evidence that the behaviour it changed is unverified.

The canonical demonstration:

def apply_discount(total: Decimal, is_member: bool) -> Decimal:
    if total > 100 and is_member:
        return total * Decimal("0.9")
    return total

def test_member_gets_discount():
    assert apply_discount(Decimal(150), True) == Decimal(135)

That test reports 100% line and 100% branch coverage of the function on many tools' definitions of the taken path. Now generate mutants:

Three of four survive. Mutant 1 says the boundary at exactly 100 is untested. Mutant 2 says no test ever supplies a non-member, so the conjunction could be a disjunction and nobody would know — a real pricing bug that gives every big-basket guest a 10% discount. Mutant 4 says the no-discount path has no test at all despite the coverage report. The tests needed:

@pytest.mark.parametrize(
    "total,is_member,expected",
    [
        (Decimal(100), True,  Decimal(100)),   # kills mutant 1: boundary is exclusive
        (Decimal(101), True,  Decimal("90.9")),
        (Decimal(150), False, Decimal(150)),   # kills mutant 2: non-member pays full
        (Decimal(50),  True,  Decimal(50)),    # kills mutant 4: fall-through returns total
    ],
)
def test_discount_rules(total, is_member, expected):
    assert apply_discount(total, is_member) == expected

This is why mutation testing is described as testing the tests. It converts a vague "are our tests any good" into a concrete, reviewable list of behaviours nobody checks.

The Operators

Mutation tools apply a fixed catalogue of small edits. Knowing the catalogue lets you predict what a run will find before you pay for it.

Removed-call mutants are the most consistently instructive. If deleting audit.log(event) or cache.invalidate(key) kills nothing, then the side effect your feature exists to produce is not under test.

Reading the Score

mutation score = killed / (generated - equivalent)

An equivalent mutant is a syntactic change with no semantic effect, so no test can possibly kill it. The classic:

i = 0
while i < n:       # mutated to: while i != n
    i += 1

With i starting at 0 and incrementing by exactly 1, the two loops are indistinguishable. Detecting equivalence in general reduces to deciding program equivalence, which is undecidable — no tool will ever do this for you. Practical consequences:

• 100% is not the goal. Chasing it means writing tests to kill mutants that cannot be killed.
• 80%+ on core domain logic is a strong result. 60% suite-wide is respectable.
• The score is a trend line, not a grade. The survivor list is the deliverable.
• Mark confirmed equivalents so they stop reappearing — a Stryker // Stryker disable next-line ... comment, or a triage file — otherwise every run re-litigates the same handful.

Set the CI threshold below your current score so the build breaks on regression, not on ambition:

{
  "mutate": ["src/pricing/**/*.ts", "src/entitlements/**/*.ts"],
  "testRunner": "jest",
  "reporters": ["html", "clear-text", "progress"],
  "thresholds": { "high": 80, "low": 65, "break": 60 }
}

Cost, and Therefore Scope

Mutation testing is expensive in a way that is structural, not incidental. The work is approximately:

runtime  ≈  mutants × (tests covering the mutated line) × per-test time

A 2,000-line module typically yields 600–1,200 mutants. Against a suite where the covering subset takes 3 seconds, that is 30–60 minutes for one module; run against a whole repo with a 40-second suite, it is comfortably a day. Tools mitigate with coverage-based test selection (run only tests that touch the mutated line), parallel workers, and early exit on first kill — but the product form never goes away.

So scope deliberately:

1. Restrict to changed files. Stryker supports git-diff scoping directly:

   npx stryker run --since=main --incremental
   

   The incremental report is cached and reused, so a PR pays only for its own diff.
2. Restrict to code where being wrong is expensive — pricing, permissions, tax, quota enforcement, state machines, retry/idempotency logic.
3. Run the broad sweep nightly or weekly, never in the per-commit gate. A mutation job in the merge path gets disabled within a month for being slow, and then you have neither the job nor the discipline.
4. Exclude the low-yield areas: generated code, migrations, serializers, dependency wiring, __repr__. Mutants there survive for uninteresting reasons and drown the signal.

# setup.cfg
[mutmut]
paths_to_mutate = src/billing/,src/entitlements/
tests_dir = tests/unit/

mutmut run
mutmut results          # summary by status
mutmut show 47          # the exact diff of survivor 47

Turning Survivors Into Tests

Triage each survivor into one of four buckets:

1. Missing assertion. The line ran, its effect was never checked. Strengthen the existing test — this is the most common and the cheapest fix.
2. Missing input class. No test supplies the input that distinguishes the mutant. Add a parametrized case, not a new test function.
3. Equivalent. Mark it, with a one-line reason. Do not write a test.
4. Dead code. Nothing can reach it. Delete the code — a survivor is sometimes telling you the branch is unreachable, which is a better finding than a test.

Bucket 1 dominates in suites that were written to a coverage target, because a coverage target rewards execution and is indifferent to assertions. That is the same failure mode test-coverage-categories addresses from the other direction: mutation testing finds the gaps empirically, coverage categories prevent them by construction.

Common Anti-Patterns

❌ Treating the mutation score as the goal — drives tests written to kill mutants rather than to describe behaviour. ✅ Use the score as a regression guard; use the survivor list as the work item.

❌ Running mutation testing on the whole repo in the PR gate — hours of CI, so it gets switched off. ✅ Diff-scoped and incremental per PR; full sweep nightly on selected packages.

❌ Chasing 100% — spends effort on undecidable equivalents. ✅ Set the break threshold just under current score and ratchet it upward.

❌ Mutating serializers, migrations, and generated code — survivors that mean nothing bury survivors that mean everything. ✅ Point it at domain logic where a wrong answer costs money or leaks data.

❌ Killing a mutant with an over-specific assertion — e.g. asserting an exact log string to kill a removed-call mutant, which then breaks on any copy edit. ✅ Assert that the effect happened and carries the right key fields.

❌ Re-triaging the same equivalents every run — the survivor list stops being read. ✅ Record equivalents inline or in a triage file with a stated reason.

❌ Interpreting a low score as "write more tests" — volume is rarely the problem. ✅ Read the diffs; the usual answer is stronger assertions in tests you already have.

Mutation Testing Checklist

• Tool wired up for the language (mutmut / cosmic-ray, Stryker, PIT)
• paths_to_mutate scoped to high-consequence domain logic
• Generated code, migrations, and serializers excluded
• Diff-scoped incremental run available for PRs
• Full sweep scheduled nightly or weekly, outside the merge gate
• Break threshold set just below the current score, ratcheted upward
• Survivor list reviewed by a human, not just the aggregate score
• Each survivor triaged: missing assertion / missing input / equivalent / dead
• Confirmed equivalent mutants recorded with a reason so they stop recurring
• Removed-call mutants specifically checked — side effects are the usual gap
• Boundary mutants closed with both-sides parametrized cases
• Score tracked over time as a trend, not reported as a grade