# Mutation Test > Mutation testing with two engines. Uses the project's NATIVE mutation runner (StrykerJS / Infection / mutmut / PIT / cargo-mutants) when one is configured — installing it on explicit consent when it is not — for a reproducible, comparable score; and an LLM-guided engine for the mutation classes native mutators cannot express: error-path removal, state mutation, async hazards, and security guard removal. Generates mutations, executes them against the relevant tests, and FIXES the tests whose gaps let a mutation survive — surviving mutants are closed in-run, not handed to another skill. Emits the cross-tool stryker-mutator.io/report.schema.json report alongside its own artifact. Flags: [path] (scope), full, --max N, --category, --runner auto|native|llm|hybrid, --break N, --no-install, --dry-run, --quick, --report-only. - Skill: `greglas75/mutation-test` (Agent Skill) - Install (CLI): `npx skillmds@latest add greglas75/mutation-test` - Raw SKILL.md: https://api.skillmd.com/api/skills/greglas75/mutation-test/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: greglas75 (https://skillmd.com/u/greglas75) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/greglas75/mutation-test --- # zuvo:mutation-test -- LLM-Guided Mutation Testing Intelligent mutation testing that targets meaningful behavioral gaps rather than random code changes. For each production file, the LLM generates mutations in 7 categories (boundary, logic, null, error, state, async, security), runs only the tests that cover that file, and closes the gaps it finds: every survivor is triaged as a real gap or an equivalent mutant, and each real gap gets the missing assertion added and re-probed in the same run. **Scope:** Production files that have associated test files. Measures how well existing tests detect real behavioral changes. **When to use:** After writing tests, before releases, when mutation score is unknown, when test suite feels shallow despite high line coverage. **Out of scope:** Writing a test suite from scratch (use `zuvo:write-tests`), fixing systematic test anti-patterns across many files (use `zuvo:fix-tests`), auditing test quality without execution (use `zuvo:test-audit`), code quality review (use `zuvo:review`). Adding the single missing assertion that a surviving mutant exposes is IN scope — that is the point of finding it. ## Argument Parsing Parse `$ARGUMENTS` as: `[path | full | continue] [--max N] [--category CATEGORY] [--runner MODE] [--break N] [--no-install] [--dry-run] [--quick] [--report-only]` | Flag | Env equivalent | Effect | |------|----------------|--------| | `[path]` | — | Scope to a specific directory or file | | `continue` | — | Resume an interrupted run from its checkpoint (Phase 3.0). Re-runs nothing already resolved. | | `full` | — | All production files that have test coverage | | `--max N` | — | Max total LLM mutations to execute (default: 50). Does not bound the native runner, which mutates exhaustively by design. | | `--category CATEGORY` | — | Only generate LLM mutations of this category: `BOUNDARY`, `LOGIC`, `NULL`, `ERROR`, `STATE`, `ASYNC`, `SECURITY` | | `--runner MODE` | `ZUVO_MUTATION_RUNNER` | `auto` (default) / `native` / `llm` / `hybrid` — see 0.1d | | `--break N` | `ZUVO_MUTATION_BREAK` | Fail the run when the final `score_triaged` is below N% (4.2c — evaluated after the 4.2b fix loop, not at 4.1). The flag overrides the env value; both are recorded with their source. | | `--no-install` | `ZUVO_MUTATION_NO_INSTALL=1` or `ZUVO_NO_INSTALL=1` | Never offer to install a mutation runner (disables the 0.1c consent gate). Detection still runs. | | `--dry-run` | — | Generate mutations and show the plan, but do not execute any. Also suppresses 0.1c entirely. | | `--quick` | — | Max 3 LLM mutations per file, max 20 total | | `--report-only` | — | Report the score; do NOT fix surviving gaps (skips 4.2b). The only way to skip the fix loop. | Flags can be combined: `zuvo:mutation-test src/services/ --max 30 --category SECURITY` Default (no arguments): **the CHANGED production files**, not the whole project — `git diff --name-only $(git merge-base HEAD )..HEAD` plus uncommitted production files, filtered to those that have tests. Then `--max 50 --runner auto`. If that set is empty, say so and stop; do NOT silently widen to everything. `full` still exists and still means every covered production file — it just has to be asked for. **Why the default moved.** Measured 2026-08-28 across the local fleet: the median mutation run is **1 minute** and normal scoped runs top out around **31 minutes**, but 32 runs exceeded an hour and the longest reached **258 minutes** — all from a main checkout, all unscoped, mutating a 2,784-file test suite to answer a question about one module. Nothing was learned in those four hours that the scoped one-minute run would not have said, and each one occupied a farm slot. A default that costs four hours when the user meant "check what I just changed" is a defect in the default, not in the farm that ran it. **Callers must pass a scope.** `zuvo:refactor` and `zuvo:write-tests` probe mutation per file (`--mutate `) and are unaffected. Any skill invoking this one must name its file set; an unscoped invocation from another skill is a bug in that skill. ## Mandatory File Loading First resolve `../../shared/includes/execution-policy.md` and `../../shared/includes/evidence-reuse.md`. Reuse the parent policy and verified evidence for a nested stage. Load only the rules needed now, with the read-once receipt protocol. Load the applicable definitions using the read-once protocol. Defer logging/retro includes until completion. ``` CORE FILES LOADED: 1. ../../rules/testing.md -- READ/MISSING 2. ../../rules/testing.md (M1-M5 + Assertion Strength + Self-Eval Evidence) -- READ/MISSING 3. ../../shared/includes/env-compat.md -- READ/MISSING 4. ../../shared/includes/run-logger.md -- READ/MISSING 5. ../../shared/includes/retrospective.md -- READ/MISSING 6. ../../shared/includes/report-output-location.md -- READ/MISSING (canonical $ZUVO_DIR for 4.3b) 7. ../../shared/includes/terminal-state.md -- READ/MISSING (HARD: no completion over a live runner) ``` **If any file is missing:** Proceed in degraded mode. Note "DEGRADED -- [file] unavailable" in the final report. ## Environment Compatibility Dispatch follows the resolved execution policy. Record actual independence and any unavailable gate; session restrictions take precedence. Read `../../shared/includes/env-compat.md` for agent dispatch patterns, path resolution, and progress tracking across all supported platforms. ## CodeSift Integration Read `../../shared/includes/codesift-setup.md` for the full initialization sequence. **Key tools for this skill:** | Phase | Task | CodeSift tool | Fallback | |-------|------|--------------|----------| | 0 | Find production files | `get_file_tree(repo, file_pattern=)` | `Glob` with detected extension | | 0 | Find test files | `get_file_tree(repo, name_pattern=)` | `Glob` with detected test pattern | | 0 | Understand file structure | `get_file_outline(repo, file_path)` | `Read` the file | | 0 | Detect complexity hotspots | `analyze_complexity(repo, top_n=20)` | Line count heuristic | | 2 | Read production code for mutation targeting | `get_symbol(repo, symbol_id)` | `Read` the file | | 2 | Batch-read multiple functions | `get_symbols(repo, symbol_ids=[...])` | Multiple `Read` calls | | 2 | Find references to identify test coverage | `find_references(repo, symbol_name)` | `Grep` for imports | --- ## Phase 0: Discovery Detect the project's test infrastructure and build the production-to-test file map. ### 0.1 Framework Detection Detect the test framework and runner from config files: | Signal | Framework | Runner command | |--------|-----------|----------------| | `jest.config.*` or `"jest"` in package.json | Jest | `npx jest` | | `vitest.config.*` or `"vitest"` in package.json | Vitest | `npx vitest run` | | `pytest.ini`, `pyproject.toml [tool.pytest]`, `conftest.py` | Pytest | `pytest` | | `phpunit.xml` | PHPUnit | `vendor/bin/phpunit` | | `_test.go` files | Go testing | `go test` | | `*_spec.rb` files | RSpec | `bundle exec rspec` | | `*.test.rs` or `#[cfg(test)]` | Rust | `cargo test` | If framework cannot be detected: ask the user for the test runner command. ### 0.1b Native mutation runner — DETECTION (read-only) Every stack this skill supports has a maintained mutation runner. A project that already configured one has a reproducible score this skill must READ rather than re-derive; the LLM engine measuring a repo that ships `stryker.conf.json` with its own hand-rolled mutants was the gap this step closes. **Detection touches nothing.** It reads config files and manifests. No install, no write, no command execution beyond a `--version` probe. | Stack | Runner | Config signals (any one) | Scoped run | Machine-readable report | |-------|--------|--------------------------|------------|--------------------------| | TS / JS | StrykerJS ≥ 10 | `stryker.conf.{json,js,mjs,cjs}`, `stryker` key in `package.json`, `@stryker-mutator/core` in devDependencies | `npx stryker run --incremental --mutate ` | `--reporters json` | | PHP | Infection ≥ 0.35 | `infection.json`, `infection.json5`, `infection.json.dist`, `infection/infection` in `composer.json` require-dev | `vendor/bin/infection --threads=max -- ` (POSITIONAL — `--filter` is deprecated since 0.34) | `logs.json` key in `infection.json` (there is no `--logger-json` CLI flag) | | Python | mutmut ≥ 3.7 | `[tool.mutmut]` in `pyproject.toml`, `[mutmut]` in `setup.cfg` | `mutmut run` — **whole project only**, see the mutmut note below | `mutmut results` | | JVM (Java, Kotlin) | PIT ≥ 1.25 | pitest plugin in `pom.xml` / `build.gradle{,.kts}` | `mvn test-compile org.pitest:pitest-maven:mutationCoverage` (the `test-compile` phase is required — a bare goal has no compiled classes to mutate) or `./gradlew pitest` | `outputFormats=XML` | | Rust | cargo-mutants ≥ 27 | `.cargo/mutants.toml`, or `cargo mutants --version` succeeds | `cargo mutants -f ` | `--json` (`mutants.out/`) | | .NET / Scala | Stryker.NET / Stryker4s | `stryker-config.{json,yaml}` | per tool docs | Stryker JSON | **mutmut 3.x cannot scope a run from the CLI.** `--paths-to-mutate` was a mutmut 2.x flag and was REMOVED in 3.x — the error message still suggests it, which is how it survives in documentation that was never re-tested. In 3.x the mutated paths come only from `[tool.mutmut] paths_to_mutate` in `pyproject.toml` / `setup.cfg`. Consequences, and they are not cosmetic: - A scoped native run on Python means **editing the project's mutmut config**, which is a write outside test files and therefore belongs to the 0.1c consent gate — not to a per-file loop. Until that is wired, treat Python native runs as **whole-project only**: run `mutmut run` as the project configures it, and take the score for the whole project rather than claiming a per-file number the tool cannot produce. - `test-mutation-probes.md`'s per-file native path is therefore **unavailable for mutmut**. That include says so explicitly; a Python project falls back to the hand-picked probes there. **Shell-quote every interpolated path.** These run commands are templates with `` placeholders, and a repository controls its own filenames — a path holding a space, a quote or a `;` becomes argument injection the moment a template is pasted into a shell. Quote the substitution (`--mutate ''`) or pass the file list as separate argv entries; never build the command by string concatenation and hand it to `bash -c`. Versions are MINIMUMS, not pins — an older configured runner is still used, with its version recorded. A runner present but BELOW the minimum is used and flagged `native_runner.state: detected (below-minimum )`; do not silently upgrade a project's tooling to satisfy a floor written here. Record on the run and in the 4.3b artifact: ``` native_runner: { name, version, config_path, state } state ∈ detected | installed | absent | declined | unsupported_stack | failed ``` ### 0.1c Consent-gated install (skipped by `--no-install`, either no-install env var, and `--dry-run`) > Mirrors the DD-3 consent gate in `skills/infra-audit/SKILL.md` — offer, consent, log with > the uninstall command, degrade loudly on decline. The wording is deliberately parallel so > the two can later be extracted into one shared include; they are not shared today, and > unifying them is a separate change to a separate skill. > > `ZUVO_MUTATION_NO_INSTALL=1` is this skill's own switch. `ZUVO_NO_INSTALL=1` is honoured > as a broader user policy, but **only this skill reads it today** — do not describe it to a > user as a machine-wide guarantee. Fires only when ALL hold: no runner detected, the stack HAS one, none of the suppressors above is set, **and the engine is not `llm`**. `--runner llm` says the run will not touch a native runner at all — prompting to install one it has already declined to use is a prompt for nothing, and a consent prompt that appears when it cannot matter is how consent stops being read. **One prompt per run — never per file.** This is the only step in this skill that writes outside test files, so it is fenced harder than anything else here: 1. **Print both commands before asking.** The exact install command AND the exact uninstall command, verbatim, so the consent is informed and reversible in one line. 2. **Dev scope only, and use the project's OWN package manager.** Never a runtime dependency, never `-g`/global for a project-scoped manager. Detect the manager from its lockfile (`pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `package-lock.json` → npm) and use the matching row — running `npm i` in a pnpm or yarn workspace writes a competing `package-lock.json` and a nested `node_modules`, which is a lasting mess left behind by a tool that was only measuring: | Manager | Install | Uninstall | |---------|---------|-----------| | npm | `npm i -D @stryker-mutator/core @stryker-mutator/-runner` (pick the plugin matching the framework detected in 0.1) | `npm rm @stryker-mutator/core @stryker-mutator/<…>-runner` | | pnpm | `pnpm add -D @stryker-mutator/core @stryker-mutator/<…>-runner` | `pnpm remove @stryker-mutator/core @stryker-mutator/<…>-runner` | | yarn | `yarn add --dev @stryker-mutator/core @stryker-mutator/<…>-runner` | `yarn remove @stryker-mutator/core @stryker-mutator/<…>-runner` | | composer | `composer require --dev infection/infection` | `composer remove --dev infection/infection` | | uv | `uv add --dev mutmut` | `uv remove --dev mutmut` | | pip | `pip install mutmut` | `pip uninstall -y mutmut` | | cargo | `cargo install cargo-mutants` — **user-global toolchain, NOT the project's `Cargo.toml`.** Say that in the prompt. | `cargo uninstall cargo-mutants` | 3. **JVM is MANUAL and is never installed.** PIT is a build-plugin edit to `pom.xml` / `build.gradle.kts` — a change to how the project builds, not a dev dependency. Print the exact snippet, do NOT edit the build file, and record `state: absent (manual wiring required)`. A skill that edits a build file to measure test quality has exceeded its mandate. 4. **Config:** generate a MINIMAL config only when none exists, only for the consented tool, and only enough to run — no opinionated thresholds, no ignore-lists. 5. **Everything written is reported.** `installed_this_run[]` in the 4.3b artifact carries `{tool, version, command, uninstall, files_touched[]}`, and the completion block prints the uninstall line. A manifest or lockfile modified without appearing there is a bug. 6. **Failure and decline are loud, never silent.** Declined → `state: declined`. Install command exits non-zero → `state: failed` with its stderr. Both fall back to the LLM engine and label the run `native: skipped ()`. A run that wanted native and got LLM must never report as if it had a native score. **Consent is a human decision and must stay one.** `--no-install` only ever makes a run *more* conservative, so an agent may set it freely; there is deliberately no flag that grants consent, because a flag an agent can type is not consent (`no-agent-typable-bypass`). Non-interactive hosts therefore take the `declined` path. ### 0.1d Engine selection (`--runner`) | Mode | Behaviour | |------|-----------| | `auto` (default) | Native if detected or installed; otherwise LLM. | | `native` | Native only. Unavailable → **ABORT** with `BLOCKED_NO_NATIVE_RUNNER` naming why. Never silently falls back — that is the whole point of asking for it. | | `llm` | The LLM engine only. This is the guaranteed floor and is unchanged from previous versions. | | `hybrid` | Native for the score, PLUS LLM mutations restricted to `ERROR`, `STATE`, `ASYNC`, `SECURITY` — the classes syntactic mutators do not generate. Report both numbers separately; **never average them**, they measure different mutant populations over the same code. | Two invariants hold in every mode: - **Phase 4.2b still runs.** A native survivor is triaged and closed in-run exactly like an LLM survivor. A native runner's HTML report is a hand-off, and handing off is the drift this skill exists to stop. - **The LLM engine is never removed.** It is the fallback for `absent`, `declined`, `failed`, and `unsupported_stack`, and it is the only engine for the four categories above. ### 0.2 File Mapping Build a map of production files to their test files. **Discovery patterns are language-aware** — use the detected framework from 0.1: | Language | Production ext | Test patterns | |----------|---------------|---------------| | TypeScript/JavaScript | `*.ts`, `*.tsx`, `*.js`, `*.jsx` | `*.test.*`, `*.spec.*`, `__tests__/*` | | Python | `*.py` | `test_*`, `*_test.py`, `tests/` | | PHP | `*.php` | `*Test.php`, `tests/` | | Go | `*.go` (non-test) | `*_test.go` | | Ruby | `*.rb` | `*_spec.rb`, `spec/` | | Rust | `*.rs` (non-test) | `#[cfg(test)]` blocks, `tests/` | For each detected language: 1. Scan for all test files using the language-specific patterns 2. For each test file, identify the production file it covers: - By import/require statements in the test - By naming convention (`foo.ts` -> `foo.test.ts`, `foo.py` -> `test_foo.py`) - By directory convention (`src/foo.ts` -> `__tests__/foo.test.ts`) 3. Build the map: `{ production_file: [test_file_1, test_file_2, ...] }` 4. Exclude production files with no test coverage (nothing to validate mutations against) If no language matches or discovery produces 0 files: ask the user for the file patterns. ### 0.3 Prioritization Order files for mutation testing by priority: 1. **Critical paths first:** Files matching keywords: `auth`, `login`, `session`, `token`, `payment`, `billing`, `charge`, `transaction`, `password`, `encrypt`, `decrypt`, `sanitize`, `validate`, `permission`, `role`, `access` 2. **High complexity:** Files with the most functions, branches, or cyclomatic complexity 3. **Recent changes:** Files with commits in the last 30 days (active development = higher risk) 4. **Everything else:** Alphabetical If `--category SECURITY` is set, promote files matching security-related keywords to the top. Output: ``` DISCOVERY Framework: [name] | Runner: [command] Native runner: [name vX.Y | none] ([detected | installed | absent | declined | failed | unsupported_stack]) Engine: [auto->native | auto->llm | native | llm | hybrid] Production files with tests: [N] Files excluded (no tests): [N] Priority order: [top 5 files listed] Scope: [path or "full project"] Max mutations: [N] ``` If `--quick`: reduce max mutations per file to 3, total to 20. --- ## Phase 1: Baseline Establish that all tests pass before introducing mutations. ### 1.1 Run Full Test Suite Execute the detected test runner command against the scoped files: ```bash # Examples: npx jest --passWithNoTests # Jest npx vitest run # Vitest pytest # Pytest go test ./... # Go ``` ### 1.2 Validate Baseline - **All tests pass:** Record the total execution time. Proceed to Phase 2. - **Any test fails:** STOP immediately. Do not proceed with mutation testing. If tests fail: ``` BASELINE FAILED [N] test(s) failing Cannot run mutation testing against a failing test suite. Suggestion: run zuvo:fix-tests to repair failing tests first. ``` ### 1.3 Send the TIER 1 loop to the farm as ONE invocation — never per-mutant, never local **Wrap the LOOP, not each mutant.** The wrapper's cost is a fixed per-invocation charge (mirror sync + queue), and Tier 1 makes N short invocations — so wrapping each one multiplies the charge by N, while wrapping the loop pays it once: rt --light bash -c '' An earlier version of this section concluded "therefore run Tier 1 locally". That was the wrong lesson from a correct measurement, and 2026-08-29 measured what it costs: 109 local test processes at 421% CPU, load 34, macOS suspending the workstation with `Dark Wake Thermal Emergency`, and a native mutation run dying ten minutes in when a concurrent worktree pulled shared `node_modules` out from under it — while the farm sat idle with ~18 free slots. Nothing runs on the workstation. Measured 2026-08-10 on the same single test file: | Invocation | Wall clock | |---|---| | `npx vitest run ` | **1.4 s** (2.2 s cold) | | `rt npx vitest run ` | **103.4 s** | ~50-75x, and a single wrapped call alone exceeds the whole run budget below. A 10-mutant plan that would finish in ~20 s locally cannot complete a single mutant through the wrapper. Wrap the LOOP once instead and the same ten mutants pay one charge. If the farm is unreachable (`rt` exits 21), this skill is not usable for that run — say so and stop; never move the loop to the workstation, and never report a wrapper timeout as a test-quality result. **Tier 2 is the opposite case, and the ban used to swallow it.** A Tier 2 pass is ONE long full-suite run — exactly the shape the global `rt` rule was written for — and the budget formula in 1.3b already accounts for it separately (`TIER2_RUNS * BASELINE_TIME`, never `PER_RUN`). The routing rule simply did not follow its own split. Measured 2026-08-17 on `rs_be`, same suite, same commit: | Tier 2 full suite | Wall clock | |---|---| | local, 4 passes in one run | 673 s + 270 s + 323 s + 312 s = **1578 s** | | farm, warm mirror + cache hit | **142–264 s** per pass (deps 2–4 s, setup < 0.8 s) | **Route Tier 2 by `BASELINE_TIME`, measured in 1.2, not by habit:** - `BASELINE_TIME >= 120s` → run Tier 2 through `rt` (`rt --light` when the suite needs no services). The one-time mirror/queue charge is amortised across a run that long, and it takes the heaviest load off the workstation that is also running the Tier 1 loop. - `BASELINE_TIME < 120s` → keep Tier 2 local. Below that the wrapper's fixed cost is a large fraction of the run and the farm wins nothing. Record which side the run took and why: `tier2_runner: "rt (baseline 187s)"` or `tier2_runner: "local (baseline 41s)"`. A Tier 2 result whose runner is unstated cannot be compared against the next run's. **Three rules that do not relax when Tier 2 goes to the farm:** 1. **Restoration stays local and unconditional.** The farm run is read-only with respect to the working tree: `rt` ships the tree as it stands (mutation applied) and the verdict comes back, but the `cp` restore in 3.2 step 4 and the hash verify in step 5 happen here, on this machine, exactly as before. Never let a remote step own the restore. 2. **A wrapper failure is not a mutation result.** `rt` exiting non-zero for a queue timeout, an evicted run, or an unreachable host means the mutation was **not measured** — it is neither killed nor survived. Follow the execution policy: keep queued work attached on the required runner. A pending run is `NOT_EXECUTED`, not a terminal refusal or a result to score around. 3. **`TEST_RAN` discipline applies to the farm too.** A farm run that is evicted or never scheduled exits 0 having run nothing, and "0 failures" from a suite that did not execute reads as a SURVIVED mutation — the single most expensive misread this skill can make. Require a summary line proving the suite executed before recording any Tier 2 verdict. **Off-tailnet, `rt` refuses with exit 21** rather than running locally. That is a routing failure, not a test result. Follow `execution-policy.md`; a project restriction on workstation execution requires waiting/reconnecting to the authorized runner. Never infer permission from a timeout. ### 1.3b Calculate Timeouts — budget per-invocation cost, not suite time **Measure the real per-invocation cost first.** Run the mapped tests for the first target file ONCE, unmutated, and record wall clock as `PER_RUN`. That number carries runner startup, transform and setup — which dominate a short targeted run and are paid again for every mutation. The full-suite baseline time does NOT predict it. ``` PER_RUN = measured wall clock of one unmutated targeted run TIER2_RUNS = expected survivors (unknown up front — budget 30% of MUTATION_COUNT) BUDGET = MUTATION_COUNT * PER_RUN * 1.5 # tier 1, +50% slack + TIER2_RUNS * BASELINE_TIME * 1.5 # tier 2 full-suite passes Per-file timeout = max(10s, 3 * PER_RUN) ``` The old formula was `3 * baseline, minimum 60s` — it budgeted three suite runs while the loop actually spends `N * (startup + short run)`. With a 5 s suite the budget was 60 s regardless of whether the plan had 5 mutants or 50, so a plan was aborted mid-way on a limit that had nothing to do with its size. Reported by a user on 2026-08-10: 7 of 10 mutants dropped, budget consumed by invocation overhead. **If `BUDGET` looks unreasonable, shrink the PLAN, not the budget** — lower `--max` or use `--quick`, and say which. Silently truncating a plan produces a mutation score computed over a sample the report presents as the whole plan. Output: ``` BASELINE Tests: [N] passing | [N] suites Baseline time: [N]s (full suite, once) Per-run cost: [N]s (one targeted run — what each mutation actually costs) Runner: rt (farm) (the whole loop in ONE invocation — see 1.3) Plan: [N] mutations -> budget [N]s ``` --- ## Phase 2: Mutation Generation For each production file (in priority order from Phase 0), generate intelligent mutations. ### 2.1 Read Production Code Read the full production file. Identify: - Functions, methods, and their signatures - Conditional branches (if/else, switch, ternary) - Guard clauses and validation - Error handling (try/catch, throw, reject) - State mutations and assignments - Async operations (await, Promise, callback) - Security-relevant code (auth checks, sanitization, access control) ### 2.2 Generate Mutations For each file, generate 5-10 mutations across these categories: | Category | Tag | Mutation type | Example | |----------|-----|--------------|---------| | Boundary | `BOUNDARY` | Off-by-one, `<` vs `<=`, `>=` vs `>`, `+1`/`-1` on limits | `i < arr.length` -> `i <= arr.length` | | Logic | `LOGIC` | `true` -> `false`, `&&` -> `\|\|`, negate condition | `if (isValid)` -> `if (!isValid)` | | Null/empty | `NULL` | Return `null` instead of value, empty array instead of data | `return users` -> `return []` | | Error path | `ERROR` | Remove try/catch, swap error types, skip validation | Remove `if (!input) throw` guard | | State | `STATE` | Remove state update, swap assignment values | `count += 1` -> `count += 0` | | Async | `ASYNC` | Remove `await`, swap resolve/reject | `await save()` -> `save()` (fire-and-forget) | | Security | `SECURITY` | Remove auth check, skip validation, remove sanitization | Remove `if (!user.isAdmin) return 403` | **Mutation quality rules:** - Each mutation must change observable behavior (not just cosmetic) - Skip trivial mutations: comments, whitespace, logging-only statements, console.log - Skip mutations in generated code, type definitions, and pure configuration - Each mutation targets one specific behavioral change - Prefer mutations at decision points (branches, guards, returns) **If `--category` is set:** Only generate mutations of the specified category. ### 2.3 Mutation Plan For each mutation, record: - `MUT-NNN`: Sequential ID - `file`: Production file path - `line`: Line number - `category`: One of BOUNDARY, LOGIC, NULL, ERROR, STATE, ASYNC, SECURITY - `original`: Original code (1-3 lines) - `mutated`: Mutated code (1-3 lines) - `rationale`: Why a test should catch this (1 sentence) - `test_files`: Which test file(s) to run Cap at `--max` total mutations (default 50). If more mutations are possible, prioritize by: 1. SECURITY mutations (most important to catch) 2. ERROR mutations (error paths are commonly under-tested) 3. BOUNDARY mutations (off-by-one errors are common and subtle) 4. LOGIC, NULL, STATE, ASYNC (remaining categories) ### 2.3b Anchor every mutation to CONTENT, not to a line number `line` is a hint that expires. It is recorded before Phase 3 runs, and by the time a mutation is applied — or re-applied on resume — the file may have moved underneath it: Phase 4.2b writes tests and can touch production code, a fix commit lands between passes, or a resumed run meets a file edited since the plan was made. Applying `mutated` at a stale line number does not fail loudly; it corrupts a DIFFERENT statement and the run then measures a mutation nobody designed. So each mutation carries an anchor that survives movement: - `symbol`: the enclosing function/method/class name (from CodeSift `get_file_outline`, or the nearest preceding definition line when unavailable). - `original_norm`: `original` with leading/trailing whitespace stripped per line and internal runs of whitespace collapsed to one space. This is what you MATCH on — never the raw text, because a formatter run would otherwise invalidate every anchor in the plan. - `occurrence`: 1-based index of `original_norm` **within `symbol`**, for the case where the same statement appears more than once in one function. - `file_sha`: `git hash-object ` at plan time. **Resolve before applying, every time:** 1. If `git hash-object ` still equals `file_sha`, the plan is current — apply at `line`. 2. Otherwise re-locate: find `symbol`, then the `occurrence`-th match of `original_norm` inside it. Exactly one match → apply there and record `anchor: relocated( → )`. 3. Zero matches, or more than one after the occurrence filter → **SKIP this mutation** and record `anchor: lost ()`. A skipped mutation is `not_run`, and per Phase 3.3 a plan that did not run in full is not a score — it must be reported as such, never averaged away. **If `--dry-run`:** Print the mutation plan and STOP. Do not execute. ``` MUTATION PLAN (--dry-run) Files: [N] Mutations: [N] total [list each mutation with ID, file, line, category, original, mutated, rationale] To execute: zuvo:mutation-test [same args without --dry-run] ``` --- ## Phase 3: Mutation Execution For each mutation in the plan, apply it, run tests, and record the result. ### 3.0 Checkpoint — write it after EVERY mutation, not at the end A mutation run is a long loop of expensive, individually-meaningful results, and until now it kept all of them in context only. Anything that ended the run — an API error, a 137, a timeout, the user stopping it — threw away every mutation already executed and left the next attempt to redo the lot. Worse, Phase 3.1 restores the file from a temp copy after each mutation, so a run killed mid-apply can leave a MUTATED file on disk with nothing on record saying so. State file: `zuvo/context/mutation-.json` (`` = first 8 of the SHA-1 of the scope argument, so concurrent runs on different scopes do not collide). ```json { "version": 1, "scope": "src/services/", "baseline": { "passed": 412, "failed": 0, "sha7": "a1b2c3d" }, "applied_to": null, "mutations": [ { "id": "MUT-001", "file": "src/x.ts", "symbol": "calcTax", "line": 88, "original_norm": "if (n > 0) {", "occurrence": 1, "file_sha": "e4f5…", "status": "killed|survived|fixed|not_run|lost", "anchor": "exact|relocated(88→91)|lost()" } ] } ``` **BEFORE anything else — including on a FRESH run — load any existing state file for this scope and honour its `applied_to`.** Writing a new plan first would overwrite the only record that a previous run died with a mutation on disk, and that mutated file then stays in the working tree, silently poisoning every later baseline in a way that looks like a real regression. `continue` is not the only path into a crashed run; the far more likely one is a user who re-runs the same command. Write it at three moments, and the middle one is the one that matters: 1. **After the plan is generated** — the full list at `status: not_run`. Only after the recovery above has run and `applied_to` is back to `null`. 2. **`applied_to: ""` BEFORE writing a mutation, back to `null` AFTER restoring it.** This is the crash-safety record: a non-null `applied_to` on startup means the previous run died with a mutation on disk. Restore that file from its temp copy (or `git checkout` it if the copy is gone and the file is tracked and otherwise clean) BEFORE doing anything else, and say so. 3. **After each mutation resolves** — its `status`, immediately, not batched at the end. **`continue` mode:** load the state file, restore any `applied_to` leftover, then execute only mutations whose `status` is `not_run`, re-resolving each anchor per 2.3b (the tree has moved since the plan — that is why you are resuming). Print what you skipped and why: ``` [MUTATION] resumed from checkpoint: 31/50 already resolved (24 killed, 5 survived, 2 fixed) [MUTATION] restored a mutated file left by the interrupted run: src/x.ts [MUTATION] 19 remaining ``` If no state file exists for the scope, say so and start fresh — never silently treat `continue` as a new full run, because the score of a partial resume and the score of a fresh run are different numbers and only one of them answers the question that was asked. ### 3.1 Safety Protocol Before starting execution: 1. **Require cleanliness only where it actually matters: the files that will be MUTATED.** `git status --porcelain -- ` must be empty. Those are the only files where the post-run hash check cannot tell a leftover mutant from an edit of yours, which is the entire hazard this step exists for. - Dirty scoped file → `BLOCKED_DIRTY_TREE`, naming it. Do not mutate it. - **Uncommitted changes ANYWHERE ELSE are not a blocker.** Record their sha256 with the rest of the pre-run snapshot and proceed. Demanding a globally clean tree blocked runs on the pipeline's OWN output — the test file `zuvo:write-tests` had just written and `memory/coverage.md` — which is a gate firing on the work it was invoked to measure. - **Never stop to ask permission to commit.** If this run authored those files, the auto-commit policy already covers them; if they are the user's, they are none of this skill's business. A question here costs a whole turn and the answer is always the same. 2. **Restoration strategy (temp copy — NOT stash):** - For each production file being mutated, copy the original to a temp location: `cp [file] /tmp/zuvo-mutation-[hash]-[filename]` - After each mutation: restore from the temp copy: `cp /tmp/zuvo-mutation-[hash]-[filename] [file]` - After ALL mutations complete (or on error): verify every original is restored, then delete temp copies - **Do NOT use `git stash`** (pop consumes the stash on first iteration) - **Do NOT use `git checkout -- [file]`** (destructive to local changes) 3. NEVER commit a mutated file. NEVER leave a mutation in place after execution. ### 3.2 Execution Loop For each mutation `MUT-NNN`: ``` 1. APPLY: Write the mutated code to the production file 2. RUN (two-tier strategy): TIER 1 — Run mapped test files first (fast, targeted): - Jest: npx jest [test_file_1] [test_file_2] --no-coverage - Vitest: npx vitest run [test_file_1] [test_file_2] - Pytest: pytest [test_file_1] [test_file_2] -x - Go: go test [package] -run [test_pattern] TIER 2 — If TIER 1 passes (mutation survived), run the FULL test suite: - This catches integration tests, black-box tests, and indirect callers - If full suite fails -> mutation KILLED (integration test caught it) - If full suite passes -> mutation truly SURVIVED --quick mode: skip TIER 2 (only mapped tests). Mark survivors as "SURVIVED (mapped tests only)" with a warning that score may be optimistic. 3. RECORD result: - Test FAILED (tier 1) -> mutation KILLED (good: direct test caught it) - Test FAILED (tier 2) -> mutation KILLED-INDIRECT (good: integration test caught it) - Test PASSED (both tiers) -> mutation SURVIVED (bad: no test caught it) - Test TIMEOUT (>per-file timeout) -> mutation TIMEOUT (counts as killed) - Test ERROR (crash/compile error) -> mutation KILLED (counts as killed) 4. RESTORE: Copy original from temp location back to production file 5. VERIFY: Diff check to confirm restoration is clean ``` **Error recovery:** If restoration fails for any reason: 1. Copy from temp file: `cp /tmp/zuvo-mutation-[hash]-[filename] [file]` 2. If temp file missing: `git checkout HEAD -- [file]` (safe: working dir was clean at start) 3. If both fail, STOP execution and alert the user **Progress tracking:** After every 10 mutations, print a progress line: ``` PROGRESS: [N]/[total] mutations executed | [killed] killed | [survived] survived ``` ### 3.2b Native runner execution (`--runner native` / `hybrid` / `auto` when one is available) The native runner replaces the 3.2 loop for the mutants it generates; it does not replace any rule around that loop. Run it scoped to the SAME file set built in 0.2, and inherit every constraint below without exception: 1. **ON THE FARM, in an isolated checkout — reversed 2026-08-29 by measurement.** This rule used to say "local, always", and following it is what produced the incident below. A native run is the single heaviest thing this repo starts: one long-lived process re-running the suite once per mutant, 730 mutants and ~24 minutes in the case that broke. What actually happened when it ran locally: the worktree's `node_modules` was shared with another worktree, the other run moved underneath it, and Stryker died ten minutes in on vanished dependencies (`mutation-testing-report-schema`, `balanced-match`) — a crash that looks like a test failure and is not. At the same moment the laptop carried 109 local test processes at 421% CPU and load 34, macOS was putting it to sleep with `Dark Wake Thermal Emergency`, and the farm sat idle with ~18 free slots. The old justification was 1.3's per-invocation wrapper overhead — real, and it does not apply here. 1.3 measures a wrapper charge paid PER SHORT CALL; a native run pays it ONCE across twenty-plus minutes, where it rounds to nothing. Send it to the farm: rt npx stryker run # or the project's own native runner **Generate `` with the helper — do NOT hand-roll it and do NOT rely on `--mutate`.** bash "$ZUVO_BASE/scripts/stryker-scoped-config.sh" \ --file --file ... # or --files-from , from the 0.2 scope set It prints `config_path`, `report_path`, `temp_dir`, `test_runner`, `coverage_analysis`, `mutate_count` and a ready `run_command`. `mutate_count` is a check, not decoration: Stryker reports an empty mutate set as a **successful run with a 100% score**, so a typo in the scope set is indistinguishable from a perfect suite unless the count is read. "Scoped Stryker config" is the most re-invented artifact in this fleet's retro log (~30 names for one thing), because it is five decisions that each fail SILENTLY: - **`--mutate` alone does not scope the run.** Stryker still loads the project config, which routinely carries a repo-wide `mutate` array, its own reporters, its own `tempDirName`. The CLI flag merges over one key; the rest still applies. - **`.stryker-tmp` is shared by every run in the repo.** Two scoped runs on one box corrupt each other's sandbox, and it surfaces as dependencies vanishing mid-run (`Cannot find module 'balanced-match'`) — which reads as a test failure and is not. This is the same incident rule 1 above was reversed for; the config is the other half of the fix. - **`coverageAnalysis: perTest` mismarks module-level ("static") mutants as SURVIVED**, because per-test coverage cannot a …(truncated)