Verify a change
Overview
This skill checks that code Claude wrote or changed is actually tested. It defines the check
before coding, runs every mutant of the changed lines with a pinned runner, and writes tests until
the tests catch them. It reports which mutants were killed and which new tests fail on the old
code, so the user can trust the change from evidence. Upstream source:
timini/agentic-verification.
When to Use This Skill
- Use after implementing a feature or fix in a Python/pytest project, before saying the work is done.
- Use when the user asks whether a change is tested.
- Use
@agentic-test-audit instead to review a whole test suite rather than one change.
How It Works
Arguments: $ARGUMENTS. --base is the git revision to compare with: HEAD (default) checks
uncommitted work, a branch such as main checks a whole branch. Other options go to the runner.
The goal is that the user can trust the change from evidence, without reading every line. A
change is verified when the tests fail if any part of it is wrong. Passing tests alone do not
show that: our study found 71% of the calls that could be deleted unnoticed sat in functions
that looked well tested.
Runner
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV verify <project-root> --base <rev> [--budget-minutes N]
$AV replay --run <run-dir> --mutant <id> --all
$AV base-check <project-root> --base <rev> [tests...]
Pass the project root (the directory pytest runs in), not a package directory or a file.
base-check runs tests against the production source as it was at --base, in a disposable
copy, and lists the tests that pass now and fail there: those detect the change.
The runner works in a disposable copy and writes only to .test-suite-audit/ in the project.
It needs Python 3.12+ and pytest-cov in the project's environment, plus pytest-randomly for
the order check. Use a Bash timeout of the budget plus 2 minutes. The last stdout line is JSON
with the verdict and report paths.
Rules
- Never weaken, skip or delete an existing test to get a verified verdict, and never edit a
test so it matches wrong behaviour. If an existing test fails, the change or the test is wrong:
find out which and tell the user.
- A new test must assert behaviour a user of the code depends on: a result, an effect, an
error. Do not assert on private helpers or mock the code under change. A test that mocks the
thing it checks passes when the feature is broken.
- Only the runner decides that a mutant is killed. Do not call a mutant equivalent yourself;
list it for the user with your reasoning.
- Do not retry a failing or flaky test until it passes. Report it; it hides real failures.
- Never stash, revert or edit production files to see a test fail on the old code. Use
base-check, which does it in a copy; an interrupted stash leaves the user's code broken.
Steps
- Define the check first. Before writing code (or, if the code exists, before testing it),
restate the success criteria in one to three lines, and pick the strongest oracle available:
- an end-to-end or public-API test of the user-visible behaviour, if there is a flow to drive;
- a property, for pure logic: round trip, invariant, idempotence, or agreement with a simple
reference implementation (Hypothesis, if the project has it or the user agrees to add it);
- a differential test, for a refactor or migration: the old version (
git show <base>:<file>)
is the oracle; compare on generated inputs and list intentional differences with a reason;
- example tests otherwise, including the edge cases the criteria name.
Write that test and see it fail before the change. If the change already exists, run
base-check to confirm the test fails on the old code.
- Keep the loop fast. While iterating, run only the tests for the area you are changing.
The verify report lists the tests covering the changed lines and how long they take.
- Run verify once the change and its tests are written. Read the verdict:
verified: every changed line runs under a test and every mutant of the changed lines was
killed. Go to step 5.
gaps: changed lines no test runs, surviving mutants, or a suite that is not green and
deterministic. Go to step 4.
incomplete: the budget ended first (rerun with a larger --budget-minutes), or no
mutant applies to the changed lines, for example a decorated function. Then the mutants
give no evidence: rely on base-check and say so.
no change: nothing under the source packages changed since --base; check the base.
- Close each gap, then rerun verify (step 3). Stop after three rounds and report what is left.
- Surviving mutant: read its patch in
mutants/<id>.patch. It shows a wrong version of your
code that the tests accept. Write a test that passes on the real code and fails on that
version, then confirm it with replay --all (killed_now: true). If the mutant changes
nothing a caller could observe, leave it and say why.
- Uncovered changed line: test it, or remove it if it is dead code.
- New test that catches none of the change's mutants (
change-weak-tests): strengthen its
assertions, or explain what it checks instead.
- Failing, intermittent or order-dependent test: report it; do not work around it. Tests
that already fail on the base revision are listed as a caveat and do not cause
gaps.
- Optional, for larger changes: review in a fresh context. If you can start a subagent, give
it only the success criteria, the diff and the test files, and ask it to write one test that
should pass and might not. Run it. A failure is a bug or a missing requirement.
- Report evidence, not a diff. Lead with the verdict, then:
- the success criteria and which tests check each one;
- mutants killed out of mutants run on the changed lines, and changed lines not covered;
- which new tests
base-check shows failing on the old code;
- tests you added, and gaps closed this way;
- what is left: surviving mutants with your reason, and anything not assessed;
- the fast check command from the report, and the path to
report.md.
Do not say the change is correct. Say what the tests now catch.
For the meaning of statuses and labels, see reading the report.
Examples
Example 1: Check uncommitted work
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV verify . --base HEAD --budget-minutes 10
Example 2: Confirm a new test kills a surviving mutant
$AV replay --run .test-suite-audit/<run-dir> --mutant <id> --all
Limitations
- Python and pytest only. Needs Python 3.12+ and
pytest-cov; the order check needs pytest-randomly.
- A killed mutant shows the tests notice that change, not that the code is correct.
- Some changes get no mutants (for example a decorated function); the verdict is then
incomplete.
- The runner decides kills; possibly equivalent mutants are left for human review.
Security & Safety Notes
- The runner is fetched with
uvx from GitHub at tag v0.3.0 (commit 07608b3b9e4e359bc0b93b3eca94bb78d67e369e). Review it before first use.
- It copies the project to a temporary directory and writes only to
.test-suite-audit/ in the project. It does not stash, revert or edit production files.
- The agent adds test files; it must not weaken, skip or delete existing tests.
Related Skills
@agentic-test-audit - audit a whole pytest suite for gaps, flaky and slow tests, and removal candidates.
1---2name: agentic-verify3description: "Check that a Python change is caught by tests: run every mutant of the changed lines and write tests until they catch them. Reports evidence instead of a diff to read."4category: testing5risk: safe6source: community7source_repo: timini/agentic-verification8source_type: community9date_added: "2026-10-16"10author: Tim Richardson11license: MIT12license_source: "https://github.com/timini/agentic-verification/blob/v0.3.0/LICENSE"13tags: [testing, pytest, mutation-testing, python, verification]14tools: [claude]15argument-hint: "[--base REV] [--budget-minutes N] [path]"16metadata:17 source: https://skillmd.com/skills/iradoweck/agentic-verify18---1920# Verify a change2122## Overview2324This skill checks that code Claude wrote or changed is actually tested. It defines the check25before coding, runs every mutant of the changed lines with a pinned runner, and writes tests until26the tests catch them. It reports which mutants were killed and which new tests fail on the old27code, so the user can trust the change from evidence. Upstream source:28[timini/agentic-verification](https://github.com/timini/agentic-verification/tree/v0.3.0/plugin/skills/verify).2930## When to Use This Skill3132- Use after implementing a feature or fix in a Python/pytest project, before saying the work is done.33- Use when the user asks whether a change is tested.34- Use `@agentic-test-audit` instead to review a whole test suite rather than one change.3536## How It Works3738Arguments: `$ARGUMENTS`. `--base` is the git revision to compare with: `HEAD` (default) checks39uncommitted work, a branch such as `main` checks a whole branch. Other options go to the runner.4041The goal is that the user can trust the change from evidence, without reading every line. A42change is verified when the tests fail if any part of it is wrong. Passing tests alone do not43show that: our study found 71% of the calls that could be deleted unnoticed sat in functions44that looked well tested.4546### Runner4748```bash49AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'50$AV verify <project-root> --base <rev> [--budget-minutes N]51$AV replay --run <run-dir> --mutant <id> --all52$AV base-check <project-root> --base <rev> [tests...]53```5455Pass the project root (the directory pytest runs in), not a package directory or a file.56`base-check` runs tests against the production source as it was at `--base`, in a disposable57copy, and lists the tests that pass now and fail there: those detect the change.5859The runner works in a disposable copy and writes only to `.test-suite-audit/` in the project.60It needs Python 3.12+ and `pytest-cov` in the project's environment, plus `pytest-randomly` for61the order check. Use a Bash timeout of the budget plus 2 minutes. The last stdout line is JSON62with the verdict and report paths.6364### Rules6566- Never weaken, skip or delete an existing test to get a verified verdict, and never edit a67 test so it matches wrong behaviour. If an existing test fails, the change or the test is wrong:68 find out which and tell the user.69- A new test must assert behaviour a user of the code depends on: a result, an effect, an70 error. Do not assert on private helpers or mock the code under change. A test that mocks the71 thing it checks passes when the feature is broken.72- Only the runner decides that a mutant is killed. Do not call a mutant equivalent yourself;73 list it for the user with your reasoning.74- Do not retry a failing or flaky test until it passes. Report it; it hides real failures.75- Never stash, revert or edit production files to see a test fail on the old code. Use76 `base-check`, which does it in a copy; an interrupted stash leaves the user's code broken.7778### Steps79801. **Define the check first.** Before writing code (or, if the code exists, before testing it),81 restate the success criteria in one to three lines, and pick the strongest oracle available:82 - an end-to-end or public-API test of the user-visible behaviour, if there is a flow to drive;83 - a property, for pure logic: round trip, invariant, idempotence, or agreement with a simple84 reference implementation (Hypothesis, if the project has it or the user agrees to add it);85 - a differential test, for a refactor or migration: the old version (`git show <base>:<file>`)86 is the oracle; compare on generated inputs and list intentional differences with a reason;87 - example tests otherwise, including the edge cases the criteria name.88 Write that test and see it fail before the change. If the change already exists, run89 `base-check` to confirm the test fails on the old code.902. **Keep the loop fast.** While iterating, run only the tests for the area you are changing.91 The verify report lists the tests covering the changed lines and how long they take.923. **Run verify** once the change and its tests are written. Read the verdict:93 - `verified`: every changed line runs under a test and every mutant of the changed lines was94 killed. Go to step 5.95 - `gaps`: changed lines no test runs, surviving mutants, or a suite that is not green and96 deterministic. Go to step 4.97 - `incomplete`: the budget ended first (rerun with a larger `--budget-minutes`), or no98 mutant applies to the changed lines, for example a decorated function. Then the mutants99 give no evidence: rely on `base-check` and say so.100 - `no change`: nothing under the source packages changed since `--base`; check the base.1014. **Close each gap**, then rerun verify (step 3). Stop after three rounds and report what is left.102 - Surviving mutant: read its patch in `mutants/<id>.patch`. It shows a wrong version of your103 code that the tests accept. Write a test that passes on the real code and fails on that104 version, then confirm it with `replay --all` (`killed_now: true`). If the mutant changes105 nothing a caller could observe, leave it and say why.106 - Uncovered changed line: test it, or remove it if it is dead code.107 - New test that catches none of the change's mutants (`change-weak-tests`): strengthen its108 assertions, or explain what it checks instead.109 - Failing, intermittent or order-dependent test: report it; do not work around it. Tests110 that already fail on the base revision are listed as a caveat and do not cause `gaps`.1115. **Optional, for larger changes: review in a fresh context.** If you can start a subagent, give112 it only the success criteria, the diff and the test files, and ask it to write one test that113 should pass and might not. Run it. A failure is a bug or a missing requirement.1146. **Report evidence, not a diff.** Lead with the verdict, then:115 - the success criteria and which tests check each one;116 - mutants killed out of mutants run on the changed lines, and changed lines not covered;117 - which new tests `base-check` shows failing on the old code;118 - tests you added, and gaps closed this way;119 - what is left: surviving mutants with your reason, and anything not assessed;120 - the fast check command from the report, and the path to `report.md`.121 Do not say the change is correct. Say what the tests now catch.122123For the meaning of statuses and labels, see [reading the report](https://github.com/timini/agentic-verification/blob/v0.3.0/plugin/skills/audit/reference/report.md).124125## Examples126127### Example 1: Check uncommitted work128129```bash130AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'131$AV verify . --base HEAD --budget-minutes 10132```133134### Example 2: Confirm a new test kills a surviving mutant135136```bash137$AV replay --run .test-suite-audit/<run-dir> --mutant <id> --all138```139140## Limitations141142- Python and pytest only. Needs Python 3.12+ and `pytest-cov`; the order check needs `pytest-randomly`.143- A killed mutant shows the tests notice that change, not that the code is correct.144- Some changes get no mutants (for example a decorated function); the verdict is then `incomplete`.145- The runner decides kills; possibly equivalent mutants are left for human review.146147## Security & Safety Notes148149- The runner is fetched with `uvx` from GitHub at tag `v0.3.0` (commit `07608b3b9e4e359bc0b93b3eca94bb78d67e369e`). Review it before first use.150- It copies the project to a temporary directory and writes only to `.test-suite-audit/` in the project. It does not stash, revert or edit production files.151- The agent adds test files; it must not weaken, skip or delete existing tests.152153## Related Skills154155- `@agentic-test-audit` - audit a whole pytest suite for gaps, flaky and slow tests, and removal candidates.