Mutation Testing

Mutation Testing with mutmut

Mutation testing assesses test suite quality by introducing small changes (mutations) to source code and verifying that tests fail. If tests don't catch a mutation, it indicates gaps in test coverage or quality.

Key Concepts

• Mutant: Modified version of code with small change
• Killed: Test fails when mutation introduced (good)
• Survived: Test passes despite mutation (test gap)
• Mutation Score: Percentage of mutants killed

Quick Start

# Install
pip install mutmut

# Run mutation testing
mutmut run

# View results
mutmut results

# Inspect specific mutant
mutmut show 1

# Apply mutation to see the change
mutmut apply 1

# Reset applied mutations
mutmut apply 0

Configuration

# setup.cfg or pyproject.toml [tool.mutmut]
[mutmut]
paths_to_mutate=src/
backup=False
runner=python -m pytest -x
tests_dir=tests/
dict_synonyms=Struct, NamedStruct

Example: Weak vs Strong Tests

# src/validators.py
def is_valid_order(order: dict) -> bool:
    if not order:
        return False
    if not order.get("items"):
        return False
    if order.get("total", 0) <= 0:
        return False
    return True

def calculate_discount(total: float, tier: str) -> float:
    if tier == "gold":
        return total * 0.2
    elif tier == "silver":
        return total * 0.1
    return 0.0

# ❌ Weak tests - mutations survive
def test_order_basic():
    order = {"items": ["a"], "total": 10}
    assert is_valid_order(order) == True  # Only tests happy path

# ✅ Strong tests - kill mutations
def test_order_null():
    assert is_valid_order(None) == False
    assert is_valid_order({}) == False

def test_order_empty_items():
    assert is_valid_order({"items": [], "total": 10}) == False

def test_order_zero_total():
    assert is_valid_order({"items": ["a"], "total": 0}) == False

def test_order_negative_total():
    assert is_valid_order({"items": ["a"], "total": -5}) == False

def test_order_valid():
    assert is_valid_order({"items": ["a"], "total": 10}) == True

def test_discount_gold():
    assert calculate_discount(100, "gold") == 20.0

def test_discount_silver():
    assert calculate_discount(100, "silver") == 10.0

def test_discount_none():
    assert calculate_discount(100, "bronze") == 0.0

Common Mutation Operators

Improving Mutation Score

When a mutant survives, add tests for:

1. Boundary conditions: Test at exact threshold values
2. Return values: Verify actual results, not just truthiness
3. All branches: Cover each conditional path
4. Edge cases: Empty, None, negative, zero

# Mutant survives: >= changed to >
def is_adult(age: int) -> bool:
    return age >= 18

# ❌ Weak: age=25 won't catch >= to > mutation
def test_adult_weak():
    assert is_adult(25) == True

# ✅ Strong: tests boundary
def test_adult_exactly_18():
    assert is_adult(18) == True

def test_not_adult_17():
    assert is_adult(17) == False

CI Integration

# .github/workflows/mutation.yml
name: Mutation Testing
on:
  pull_request:
    paths: ['src/**']

jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - run: pip install mutmut pytest
      - run: mutmut run --CI
      - run: mutmut results

Best Practices

• Target critical business logic, not all code
• Aim for 80%+ mutation score on important modules
• Review survived mutants to improve tests
• Test boundaries and edge cases thoroughly
• Verify actual values, not just code execution

Source: jaredmcfarland/mixpanel_data — distributed by TomeVault.