Mutation Testing

Mutation Testing

Add a third validation layer to Outside-In TDD workflow. Acceptance tests verify WHAT (observable behavior), Domain tests verify HOW (business rules), mutation testing verifies tests actually catch bugs.

Core Concept

Mutation testing introduces deliberate bugs (mutants) into source code, then runs the test suite. If tests fail, the mutant is killed ✓. If tests pass despite the bug, the mutant survives ✗ (test gap found).

Source code → introduce mutation → run tests
                                     ├── tests FAIL → mutant killed ✓
                                     └── tests PASS → mutant survived ✗

A project with 100% code coverage can still have a 60% mutation score — meaning 40% of introduced bugs go undetected.

When to Use

Run mutation testing after the relevant test baseline is green:

1. ✅ Core behavior tests pass
2. ✅ Rule-focused tests pass
3. 🧬 Mutation testing — verify tests detect regressions

Never run on red baseline — mutation assumes tests work correctly first.

Approach for .NET/C#

Primary: Stryker.NET (Recommended)

For .NET projects, Stryker.NET is the established mutation framework with excellent C# support. No config file needed — all options are passed via CLI.

Install (only if not already available):

# Check first — if this succeeds, skip installation entirely. Do NOT manipulate PATH.
dotnet stryker --version

# Only run if the above command fails (tool not found)
dotnet tool install -g dotnet-stryker

Run on changed code only (default workflow — use after every story):

# Mutate only files changed since main — fast, targeted
dotnet stryker \
  --project src/YourProject.Domain/YourProject.Domain.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --mutate "**/*.cs" --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  --since:main \
  --break-at 100 \
  -r json

--since:main — only mutants within git-diff vs main are tested. Unchanged code produces no result. Fast.

Run full business logic (use before merge):

dotnet stryker \
  --project src/YourProject.Core/YourProject.Core.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --mutate "src/YourProject.Core/**/*.cs" \
  --mutate "src/YourProject.Application/**/*.cs" \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  --break-at 100 \
  --threshold-high 90 --threshold-low 80 \
  -r json -r cleartext

Cumulative baseline — full picture after incremental runs:

# --with-baseline combines --since with a persistent baseline report
# Use this in CI to keep a full history while only re-testing changed code
dotnet stryker \
  --project src/YourProject.Core/YourProject.Core.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --with-baseline:main \
  --break-at 100 \
  -r json

--with-baseline = --since + saves/loads a baseline report. Gives a complete score even when only changed files were re-tested.

Alternative: Custom Mutation Tool (For Specific Needs)

Build a custom tool only when:

• Stryker doesn't cover domain-specific mutation patterns
• You need tight integration with custom test infrastructure
• Performance optimization requires targeted mutation scope

Architecture (3 modules):

1. Mutations — rules table (+ → -, true → false, >= → >)
2. Runner — source-to-test mapping, targeted test execution
3. Core — orchestration: apply mutation → run tests → restore → report

For full custom tool reference, see Uncle Bob's empire-2025 mutation testing.

Core Mutation Categories

Workflow

Universal prerequisite — applies to every step, every scenario: Before any mutation activity (first run, CI setup, killing survivors, analyzing reports), the test suite for the affected scope must be green. If tests are failing, fix them first. Mutation results on a red baseline are meaningless — failing tests cannot kill mutants they already can't run.

Step 1: Verify Prerequisites

Before running mutation testing, confirm:

• ✅ Baseline tests are green for the mutated scope
• ✅ Meaningful unit tests exist (mutation runs against unit tests)
• ✅ No uncommitted changes (mutations modify source temporarily)
• ✅ Tests are fast (< 100ms each) — slow tests = slow mutation runs

Step 2: Set Mutation Scope

Target critical business logic first:

• Domain policies, decision engines, pricing/risk calculators
• Application orchestration with complex conditionals
• Validation rules and boundary behavior

Exclude from mutation:

• DTOs, data structures without logic
• Infrastructure (repositories, adapters)
• Configuration, DependencyInjection files
• Generated code, marker interfaces

Progressive scoping:

Step 3: Run Mutations

During development (fast, on changed code only):

dotnet stryker \
  --project src/YourProject.Core/YourProject.Core.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --since:main \
  --break-at 100 \
  -r json

Before merge (full business logic scope):

dotnet stryker \
  --project src/YourProject.Core/YourProject.Core.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --mutate "src/YourProject.Core/**/*.cs" \
  --mutate "src/YourProject.Application/**/*.cs" \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  --break-at 100 \
  -r json -r cleartext

Metrics:

• Total mutants generated
• Mutants killed (tests caught the bug ✓)
• Mutants survived (test gap ✗)
• Mutation score: (killed / total) × 100

--since note: unchanged files produce no result — this is expected. Survivors and kills only apply to the diff scope.

Expected duration: --since run: ~1-3 min. Full run: ~5-15 min (depends on test suite speed).

Step 4: Analyze Survivors

Query survivors directly from the JSON report — do not read the full file:

jq '[.files | to_entries[] | {file: .key, survivors: [.value.mutants[] | select(.status == "Survived") | {mutator: .mutatorName, line: .location.start.line, replacement: .replacement}]}] | map(select(.survivors | length > 0))' \
  StrykerOutput/$(ls -t StrykerOutput | head -1)/reports/mutation-report.json

For each surviving mutant:

1. Read the mutation — what was changed? (e.g., >= → >, removed if branch)
2. Identify unguarded behavior — which business rule isn't tested?
3. Categorize:
   • Real gap — behavior change not caught by tests
   • Equivalent mutant — mutation doesn't change observable behavior

Equivalent mutant examples:

• x = x + 0 changed to x = x + 1 (dead code)
• Logging statements removed (no observable effect)
• Defensive null checks when value is guaranteed non-null by type

After classifying survivors, always include a targeted re-run command scoped to the files that contain real gaps — this confirms kills after you write new tests and gives reviewers a runnable artifact:

dotnet stryker \
  --project <YourProject.Domain.csproj> \
  -tp <path/to/YourProject.UnitTests/YourProject.UnitTests.csproj> \
  --mutate "**/<FileWithRealGap>.cs" \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  --break-at 100 \
  -r cleartext

Step 5: Kill Surviving Mutants

For each real survivor (not equivalent):

1. Write a new test targeting the unguarded behavior
2. Run test against mutated code (using Stryker's mutation operator):
   • Expected: test FAILS (catches the bug)
3. Run test against original code:
   • Expected: test PASSES
4. Re-run Stryker to confirm kill

Example:

Survivor: if (age >= 18) mutated to if (age > 18) → survived

// New test to kill the boundary mutant
[Fact]
public void WhenDriverIsExactly18_ShouldBeEligible()
{
    var policy = new EligibilityPolicy();
    var driver = new DriverInfo(Age: 18, LicenseYears: 1);
    var vehicle = new VehicleInfo(Type: "sedan", Age: 1);

    var result = policy.Evaluate(driver, vehicle);

    Assert.True(result.IsEligible); // Fails if mutant uses `age > 18`
}

Step 6: Report & Document

Present summary with before/after metrics:

Mutation Testing Report — Core Business Layer
═══════════════════════════════════════
Scope: YourProject.Core.Policies

Score:  68% → 82% (after killing survivors)
Killed:   82 / 100
Survived:  18 → 10

New tests added: 8
- Boundary tests for age/experience thresholds: 4
- Edge cases for vehicle type combinations: 3
- Null/empty validation: 1

Remaining survivors (equivalent mutants — documented):
- EligibilityPolicy.cs:L42 — removed log statement (no observable effect)
- DriverAge.cs:L15 — defensive null check (guaranteed non-null by type)

Document legitimate survivors in code comments or architecture decision records.

Mutation Score Targets

Set thresholds based on team policy and risk profile. Common practice is to start with a progressive threshold and tighten it over time.

Equivalent mutants are the only legitimate exception — document them explicitly.

Progressive Threshold Strategy

CI/CD integration:

# In CI pipeline - fail build if below 100%
dotnet stryker --break-at [team-threshold]

When the CI gate fails, it means survivors remain. Do not raise the threshold to pass — investigate each survivor first. Classify them as real gap (write a test) or equivalent mutant (document). Only equivalent mutants are an acceptable reason to adjust the threshold.

Integration with Outside-In TDD

Mutation testing is the third validation layer:

1. Gherkin scenarios (WHAT)       → Acceptance tests
2. Business rules (HOW)            → Domain tests  
3. Test effectiveness (REAL?)     → Mutation testing

Workflow integration:

1. Write Gherkin scenario (outside-in-tdd)
2. RED → validate → SYNTHESIZE GREEN (red-synthesize-green)
3. After story complete: Run mutation testing on affected business logic modules
4. Kill critical survivors before merge

Anti-Patterns

"Let me mutate before tests are green"

No. Fix failing tests first. Mutation assumes a green baseline.

"100% is unrealistic"

Aggressive targets can be appropriate for critical logic, but thresholds are a policy decision. Equivalent mutants remain the only valid exception to survivor cleanup.

"Mutate everything including Infrastructure"

Never mutate repositories, adapters, and pure plumbing. Focus on business logic first.

"Run mutations on every commit"

Too slow. Run on feature completion or weekly. CI runs only on PR.

"Ignore all survivors as equivalent"

Rationalization. Most survivors are real gaps. Investigate each one.

"Chase the score, not the quality"

Mutation score is a signal, not the goal. Focus on killing mutants that represent real behavioral gaps.

Common Mistakes

Tools & Commands

Install / update Stryker.NET:

dotnet tool install -g dotnet-stryker
dotnet tool update -g dotnet-stryker

On changed code only (fast — during development):

dotnet stryker \
  --project <YourProject.Domain.csproj> \
  -tp <path/to/YourProject.UnitTests/YourProject.UnitTests.csproj> \
  --since:main \
  --break-at 100 \
  -r json

Full business logic scope (before merge):

dotnet stryker \
  --project <YourProject.Domain.csproj> \
  -tp <path/to/YourProject.UnitTests/YourProject.UnitTests.csproj> \
  --mutate "src/<YourProject>.Domain/**/*.cs" \
  --mutate "src/<YourProject>.Application/**/*.cs" \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  --break-at 100 \
  --threshold-high 100 --threshold-low 100 \
  -r json -r cleartext

Cumulative baseline in CI (full picture + incremental speed):

dotnet stryker \
  --project <YourProject.Domain.csproj> \
  -tp <path/to/YourProject.UnitTests/YourProject.UnitTests.csproj> \
  --with-baseline:main \
  --break-at 100 \
  -r json

Scope to a specific file or feature (debug a survivor):

dotnet stryker \
  --project <YourProject.Domain.csproj> \
  -tp <path/to/YourProject.UnitTests/YourProject.UnitTests.csproj> \
  --mutate "**/<TargetFile>.cs" \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  --break-at 100 \
  -r cleartext

Inspect JSON report:

jq '.' StrykerOutput/**/reports/mutation-report.json | head -n 120

Key CLI flags reference:

References

• Stryker.NET Documentation
• Stryker.NET Configuration
• Uncle Bob's Mutation Testing Plan
• Mutation Testing Patterns

Integration

REQUIRED BACKGROUND: superpowers-whetstone:outside-in-tdd — defines the two test streams (Application + Domain) REQUIRED BACKGROUND: superpowers-whetstone:red-synthesize-green — TDD cycle that produces tests to mutate

WORKFLOW:
Run mutation testing after story completion, before PR/merge. Use as quality gate, not coverage metric.