Chunked Mutation Testing

Chunked Mutation Testing (Go + gremlins)

A full gremlins unleash on a large Go module can crash: OOM, timeout, or a dead

container — and the work with it. Run it in chunks instead: one gremlins process per

chunk, file-level selection via --exclude-files. Small, resumable, crash-isolated.

Hard Rules

1. Never two gremlins at once. Check pgrep -a gremlins before every start.

2. One process per chunk. A crash only loses the current chunk. Mark it, restart, continue.

3. Dry-run first (analysis only, no tests). Dry-run crashes → split the chunk, don't tune flags.

4. Wall-clock guard: wrap real runs in timeout. Hit → split the chunk.

5. Drossel: --workers 2 --test-cpu 2 --timeout-coefficient 2. Raise only when a large chunk

   is slow AND memory stays comfortable (free -m after the run).

Command Template

# 1) Chunk files (one or more production .go files, space-separated)

CHUNK_LIST="pkg/server/handler.go"



# 2) Exclusion regex: every other production .go file in the module

#    NOTE: awk, not `grep -vxF -f <(...)` — grep drops the whole pattern file when the

#    input is a process substitution fd (verified: chunk files leak into the excludes)

EXCLUDES=$(find . -path './.git' -prune -o -name '*.go' ! -name '*_test.go' -print \

  | awk -v chunk="$CHUNK_LIST" 'BEGIN{n=split(chunk,a," "); for(i in a) keep[a[i]]=1} !($0 in keep) {print "^"$0"$"}' \

  | paste -sd'|')



# 3) Dry-run (no tests executed; expect RUNNABLE > 0, exit 0)

gremlins unleash --dry-run --exclude-files "$EXCLUDES" --workers 2 --test-cpu 2 --timeout-coefficient 2



# 4) Real run — only if dry-run exited 0

mkdir -p gremlins-results

timeout 45m gremlins unleash --exclude-files "$EXCLUDES" \

  --workers 2 --test-cpu 2 --timeout-coefficient 2 \

  -o gremlins-results/<chunk-name>.json

gremlins-results/ should be git-ignored. For a package-level chunk, list all non-test

.go files of that package. Adapt the find root if the module has other source dirs.

Settings

• Active mutators (gremlins defaults): arithmetic-base, conditionals-boundary,

  conditionals-negation, increment-decrement, invert-negatives

• Disabled by default, rarely worth enabling wholesale: invert-assignments, invert-bitwise,

  invert-bwassign, invert-logical, invert-loopctrl, remove-self-assignments

• Build-tag test suites (e.g. integration tests) never run without --tags — that's usually

  intended; say so explicitly if the caller expects them counted

• As a release gate: enable more mutators + --threshold-efficacy / --threshold-mcover

• --diff <ref> mutates only changed files — cheaper alternative for pre-PR checks

Reading Results

• Only RUNNABLE mutants are tested; NOT COVERED are ignored by real runs.

• Trust a report only if its -o JSON exists — a "completed" stdout line with no artifact

  is a silent no-op. Verify: JSON exists, count statuses from it, cross-check the total

  against the dry-run's RUNNABLE count.

• LIVED mutants → tests too weak at that spot; fix tests, re-run the chunk.

• free -m after a run documents the memory picture for the next calibration decision.

Common Mistakes

| Mistake | Consequence / Fix |

|---|---|

| Full run on a large module | OOM/timeout, container dies — chunk instead |

| Chunks in parallel | Resource exhaustion; strictly sequential, one process at a time |

| Trusting "done" without the JSON | Silent no-op; always verify artifacts |

| Excludes via grep -f <(...) | Pattern file not applied; chunk files excluded too — use the awk form |

| Splitting by Go subpackage | Mutants live in packages, but a huge single package can't shrink that way — file level is the boundary |

| Tuning workers/timeout when a run dies | Splitting the chunk is the reliable lever; flags only shift the cliff |