Mutation Test Runner
Overview
Execute mutation testing to evaluate the effectiveness of a test suite by systematically introducing small code changes (mutants) and checking whether existing tests detect them. A killed mutant means the tests caught the change; a surviving mutant reveals a testing gap.
Prerequisites
- Mutation testing framework installed (Stryker, mutmut, PITest, or go-mutesting)
- Existing test suite with reasonable pass rate (all tests must pass before mutation testing)
- Source code with functions and logic suitable for mutation (conditionals, arithmetic, return values)
- Sufficient CI resources (mutation testing runs the test suite once per mutant -- CPU-intensive)
- Configuration file for the mutation tool specifying target files and test commands
Instructions
- Verify the existing test suite passes completely:
- Run the full test suite and confirm 100% pass rate.
- Fix any failing or skipped tests before proceeding.
- Mutation testing is meaningless if the baseline tests are broken.
- Configure the mutation testing tool:
- Stryker: Create
stryker.config.mjs with mutate patterns, test runner, and thresholds.
- mutmut: Configure
setup.cfg or pyproject.toml with [mutmut] section.
- PITest: Add Maven/Gradle plugin with target classes and test configurations.
- Select target files for mutation:
- Focus on business logic modules (not configuration, constants, or type definitions).
- Exclude auto-generated code, third-party wrappers, and test utilities.
- Start with a small scope (one module) to validate setup before expanding.
- Run the mutation testing suite:
- Execute
npx stryker run, mutmut run, or mvn pitest:mutationCoverage.
- Monitor progress -- expect long execution times (10-100x normal test runtime).
- Use incremental mode if available to skip already-tested mutants.
- Analyze the mutation report:
- Killed mutants: Tests detected the change -- indicates strong test coverage.
- Survived mutants: Tests did not catch the change -- indicates a testing gap.
- Timed out mutants: Mutation caused an infinite loop -- generally acceptable.
- No coverage mutants: The mutated code is not exercised by any test.
- For each surviving mutant, determine the appropriate action:
- Write a new test that specifically catches the mutation.
- Or determine the mutation is equivalent (functionally identical to original) and mark as ignored.
- Set mutation score thresholds (recommended: 80% kill rate) and integrate into CI as a quality gate.
Output
- Mutation testing report (HTML or JSON) with killed/survived/timed-out counts
- Mutation score percentage (killed / total non-equivalent mutants)
- Surviving mutant inventory with file, line, mutation type, and suggested test
- New test cases written to kill surviving mutants
- CI configuration with mutation score threshold enforcement
Error Handling
| Error |
Cause |
Solution |
| Mutation run takes hours |
Too many files in scope or slow test suite |
Narrow mutate scope to critical modules; use --incremental mode; parallelize with --concurrency |
| All mutants survive |
Tests only check for truthiness, not specific values |
Strengthen assertions -- use toBe(42) instead of toBeTruthy(); add boundary checks |
| Equivalent mutant false positive |
Mutation produces functionally identical code (e.g., x >= 0 vs x > -1) |
Mark as equivalent in config; ignore in score calculation; document rationale |
| Out of memory during run |
Too many concurrent mutation workers |
Reduce --concurrency setting; increase Node.js --max-old-space-size; reduce shard size |
| Stryker "initial test run failed" |
Test suite does not pass cleanly before mutations begin |
Fix all failing tests first; ensure npm test exits 0; check test runner configuration |
Examples
Stryker configuration for TypeScript project:
// stryker.config.mjs
export default {
mutate: ['src/**/*.ts', '!src/**/*.d.ts', '!src/**/index.ts'],
testRunner: 'jest',
jest: { configFile: 'jest.config.ts' },
reporters: ['html', 'clear-text', 'progress'],
thresholds: { high: 80, low: 60, break: 50 },
concurrency: 4,
timeoutMS: 10000, # 10000: 10 seconds in ms
};
Example surviving mutant and fix:
Mutant: src/utils/discount.ts:15 -- ConditionalExpression
Original: if (total > 100)
Mutant: if (total >= 100)
Status: SURVIVED
Fix -- add boundary test:
it('does not apply discount at exactly 100', () => {
expect(calculateDiscount(100)).toBe(0);
});
it('applies discount above 100', () => {
expect(calculateDiscount(101)).toBe(10.1);
});
mutmut for Python:
# Run mutation testing
mutmut run --paths-to-mutate=src/ --tests-dir=tests/
# View surviving mutants
mutmut results
# Inspect a specific mutant
mutmut show 42
Resources
1---2name: running-mutation-tests3description: Execute mutation testing to evaluate test suite effectiveness. Use when performing specialized testing. Trigger with phrases like "run mutation tests", "test the tests", or "validate test effectiveness".4license: MIT5---6# Mutation Test Runner
7
8## Overview
9
10Execute mutation testing to evaluate the effectiveness of a test suite by systematically introducing small code changes (mutants) and checking whether existing tests detect them. A killed mutant means the tests caught the change; a surviving mutant reveals a testing gap.
11
12## Prerequisites
13
14- Mutation testing framework installed (Stryker, mutmut, PITest, or go-mutesting)
15- Existing test suite with reasonable pass rate (all tests must pass before mutation testing)
16- Source code with functions and logic suitable for mutation (conditionals, arithmetic, return values)
17- Sufficient CI resources (mutation testing runs the test suite once per mutant -- CPU-intensive)
18- Configuration file for the mutation tool specifying target files and test commands
19
20## Instructions
21
221. Verify the existing test suite passes completely:
23 - Run the full test suite and confirm 100% pass rate.
24 - Fix any failing or skipped tests before proceeding.
25 - Mutation testing is meaningless if the baseline tests are broken.
262. Configure the mutation testing tool:
27 - Stryker: Create `stryker.config.mjs` with `mutate` patterns, test runner, and thresholds.
28 - mutmut: Configure `setup.cfg` or `pyproject.toml` with `[mutmut]` section.
29 - PITest: Add Maven/Gradle plugin with target classes and test configurations.
303. Select target files for mutation:
31 - Focus on business logic modules (not configuration, constants, or type definitions).
32 - Exclude auto-generated code, third-party wrappers, and test utilities.
33 - Start with a small scope (one module) to validate setup before expanding.
344. Run the mutation testing suite:
35 - Execute `npx stryker run`, `mutmut run`, or `mvn pitest:mutationCoverage`.
36 - Monitor progress -- expect long execution times (10-100x normal test runtime).
37 - Use incremental mode if available to skip already-tested mutants.
385. Analyze the mutation report:
39 - **Killed mutants**: Tests detected the change -- indicates strong test coverage.
40 - **Survived mutants**: Tests did not catch the change -- indicates a testing gap.
41 - **Timed out mutants**: Mutation caused an infinite loop -- generally acceptable.
42 - **No coverage mutants**: The mutated code is not exercised by any test.
436. For each surviving mutant, determine the appropriate action:
44 - Write a new test that specifically catches the mutation.
45 - Or determine the mutation is equivalent (functionally identical to original) and mark as ignored.
467. Set mutation score thresholds (recommended: 80% kill rate) and integrate into CI as a quality gate.
47
48## Output
49
50- Mutation testing report (HTML or JSON) with killed/survived/timed-out counts
51- Mutation score percentage (killed / total non-equivalent mutants)
52- Surviving mutant inventory with file, line, mutation type, and suggested test
53- New test cases written to kill surviving mutants
54- CI configuration with mutation score threshold enforcement
55
56## Error Handling
57
58| Error | Cause | Solution |
59|-------|-------|---------|
60| Mutation run takes hours | Too many files in scope or slow test suite | Narrow `mutate` scope to critical modules; use `--incremental` mode; parallelize with `--concurrency` |
61| All mutants survive | Tests only check for truthiness, not specific values | Strengthen assertions -- use `toBe(42)` instead of `toBeTruthy()`; add boundary checks |
62| Equivalent mutant false positive | Mutation produces functionally identical code (e.g., `x >= 0` vs `x > -1`) | Mark as equivalent in config; ignore in score calculation; document rationale |
63| Out of memory during run | Too many concurrent mutation workers | Reduce `--concurrency` setting; increase Node.js `--max-old-space-size`; reduce shard size |
64| Stryker "initial test run failed" | Test suite does not pass cleanly before mutations begin | Fix all failing tests first; ensure `npm test` exits 0; check test runner configuration |
65
66## Examples
67
68**Stryker configuration for TypeScript project:**
69
70```javascript
71// stryker.config.mjs
72export default {
73 mutate: ['src/**/*.ts', '!src/**/*.d.ts', '!src/**/index.ts'],
74 testRunner: 'jest',
75 jest: { configFile: 'jest.config.ts' },
76 reporters: ['html', 'clear-text', 'progress'],
77 thresholds: { high: 80, low: 60, break: 50 },
78 concurrency: 4,
79 timeoutMS: 10000, # 10000: 10 seconds in ms
80};
81```
82
83**Example surviving mutant and fix:**
84
85```
86Mutant: src/utils/discount.ts:15 -- ConditionalExpression
87 Original: if (total > 100)
88 Mutant: if (total >= 100)
89 Status: SURVIVED
90
91Fix -- add boundary test:
92it('does not apply discount at exactly 100', () => {
93 expect(calculateDiscount(100)).toBe(0);
94});
95it('applies discount above 100', () => {
96 expect(calculateDiscount(101)).toBe(10.1);
97});
98```
99
100**mutmut for Python:**
101
102```bash
103# Run mutation testing
104mutmut run --paths-to-mutate=src/ --tests-dir=tests/
105
106# View surviving mutants
107mutmut results
108
109# Inspect a specific mutant
110mutmut show 42
111```
112
113## Resources
114
115- Stryker Mutator: https://stryker-mutator.io/
116- mutmut (Python): https://github.com/boxed/mutmut
117- PITest (Java): https://pitest.org/
118- go-mutesting: https://github.com/zimmski/go-mutesting
119- Mutation testing theory: https://en.wikipedia.org/wiki/Mutation_testing
gabrielmoreira