Audit a test suite
Overview
This skill measures whether a Python test suite is a good check. It asks whether the suite would
notice broken code, whether it is fast and stable enough to run on every change, and which tests
cost time without adding detection. A pinned runner does the measuring; the skill reports and
does not change tests. Upstream source:
timini/agentic-verification.
When to Use This Skill
- Use when the user asks to audit, review, speed up, prune or strengthen their pytest tests.
- Use when the user asks which tests are useful, flaky or slow.
- Use
@agentic-verify instead to check a single change.
How It Works
Arguments: $ARGUMENTS. The first word is the mode: quick (default) or deep. Anything
else goes to the runner. To check a single change rather than the whole suite, use the
agentic-verify skill instead.
An agent can only be trusted as far as the checks it runs. This skill measures whether a test
suite is a good check: whether it would notice broken code, whether it is fast and stable enough
to run on every change, and which tests cost time without adding detection.
Runner
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV quick|deep <path> [--budget-minutes N]
$AV prune-check <path> --tests-file <file> --from-run <run-dir>
$AV replay --run <run-dir> --mutant <id>
The runner works in a disposable copy and writes only to .test-suite-audit/ in the project;
suggest adding that to .gitignore. It needs Python 3.12+ and pytest-cov in the project's
environment, and pytest-randomly for order checks. Use a Bash timeout of the budget plus 2
minutes. The last stdout line is JSON with the report paths.
Rules
- The runner measures; you interpret. Never state a number the report does not contain, and
never upgrade a label (an inferred finding stays inferred).
- Report-only. Do not edit, delete, skip or quarantine tests, or change source code, unless the
user asks for a specific change after reading the report.
- Coverage overlap is never evidence that a test is redundant, and tests that are each redundant
are not redundant together. Only a removal set that
prune-check did not refute may be
presented as removable, and even then as "not refuted", not "safe".
- A surviving mutant is a candidate gap, not proof of a bug. Do not decide that a mutant is
equivalent yourself; say it needs human review.
- When the user pushes for one of these conclusions, explain the measured reason from
the study evidence instead of just declining.
Steps
- Check the target. Confirm it is a pytest project. Look for
test-suite-audit.toml; if
absent, the runner discovers source packages, test directories and the interpreter
(.venv/bin/python, then python3). If discovery looks wrong (for example a monorepo),
write a test-suite-audit.toml only after asking. Keys: python, source, tests,
pytest_args, env, workers, protected (node ID prefixes or marker names of tests that
must never be proposed for removal). If pytest-cov is missing, say coverage will be "not
assessed" and offer the install command; do not install anything without asking.
- State the budget before running. Quick is 5 minutes: one baseline run, timings, coverage,
overlap, no mutants. Deep is 30 minutes by default: three file-order and three random-order
runs, coverage, every deletion mutant, then a seeded sample of classical mutants until the
budget ends. In our benchmark, 12 to 14 minutes completed deep audits of projects up to about
3,300 executable lines; dateutil (about 3,600) and click (about 5,000) needed more.
- Run the audit and read
report.json (full evidence) and report.md. See
reading the report for statuses, labels and the detection rate.
- If the deep audit proposed removal candidates, treat them as an upper bound: in our
study such sets covered about 80% of each suite and every one lost kills on held-out mutants.
Run
prune-check on them with
--from-run <run-dir>, so the check uses mutants the set was not chosen from. Give it at
least 10 minutes. Report its verdict: refuted (with the kills or lines lost) or not refuted
(with the number of killed mutants checked and the upper bound it gives).
If it is refuted, keep the tests it names as sole killers, and say the rest is still
unproven. Never present a refuted set as removable.
- Summarise under three headings, findings in the report's order, each with its label,
evidence and next validation step:
- Does it catch broken code? Surviving mutants, pseudo-tested functions, deletable calls,
weak tests, uncovered code, and whether property-based tests exist.
- Can an agent run it on every change? Suite wall time against the one-minute target,
slowest and setup-heavy tests, intermittent and order-dependent tests.
- Is it bloated? The removal set and its prune-check verdict, with the time it would save.
Overlap pairs are listed only as things to look at.
Give the reproduction command for at least one mutant finding. List what was not assessed.
Link
report.md.
- Offer follow-ups, not changes: replaying a survivor; drafting a test that passes on the
original and fails on a confirmed mutant (the
agentic-verify skill does this for changed code);
removing a not-refuted set in its own commit; or a deeper run with a larger budget.
If the run is partial (budget or interruption), say so first and explain which steps are missing.
Examples
Example 1: Quick audit
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV quick . --budget-minutes 5
Example 2: Deep audit, then check a removal set on held-out mutants
$AV deep . --budget-minutes 30
$AV prune-check . --tests-file removal.txt --from-run .test-suite-audit/<run-dir>
Limitations
- Python and pytest only. Needs Python 3.12+ and
pytest-cov; order checks need pytest-randomly.
- A surviving mutant is a candidate gap, not proof of a bug.
- A
not refuted removal set is not proof that removal is safe.
- Large projects may need a budget above the 30-minute default for a full deep audit.
Security & Safety Notes
- The runner is fetched with
uvx from GitHub at tag v0.3.0 (commit 07608b3b9e4e359bc0b93b3eca94bb78d67e369e). Review it before first use.
- It works in a temporary copy of the project and writes only to
.test-suite-audit/.
- Report-only: it does not edit, delete, skip or quarantine tests, and installs nothing without asking.
Related Skills
@agentic-verify - check one change by running every mutant of the changed lines.
1---2name: agentic-test-audit3description: "Audit a pytest suite: surviving mutants and untested code, flaky, order-dependent and slow tests, and removal candidates checked on held-out mutants. Report-only."4category: testing5risk: safe6source: community7source_repo: timini/agentic-verification8source_type: community9date_added: "2026-10-16"10author: Tim Richardson11license: MIT12license_source: "https://github.com/timini/agentic-verification/blob/v0.3.0/LICENSE"13tags: [testing, pytest, mutation-testing, python, flaky-tests, test-audit]14tools: [claude]15argument-hint: "quick | deep [--budget-minutes N] [--max-mutants N] [path]"16metadata:17 source: https://skillmd.com/skills/iradoweck/agentic-test-audit18---1920# Audit a test suite2122## Overview2324This skill measures whether a Python test suite is a good check. It asks whether the suite would25notice broken code, whether it is fast and stable enough to run on every change, and which tests26cost time without adding detection. A pinned runner does the measuring; the skill reports and27does not change tests. Upstream source:28[timini/agentic-verification](https://github.com/timini/agentic-verification/tree/v0.3.0/plugin/skills/audit).2930## When to Use This Skill3132- Use when the user asks to audit, review, speed up, prune or strengthen their pytest tests.33- Use when the user asks which tests are useful, flaky or slow.34- Use `@agentic-verify` instead to check a single change.3536## How It Works3738Arguments: `$ARGUMENTS`. The first word is the mode: `quick` (default) or `deep`. Anything39else goes to the runner. To check a single change rather than the whole suite, use the40`agentic-verify` skill instead.4142An agent can only be trusted as far as the checks it runs. This skill measures whether a test43suite is a good check: whether it would notice broken code, whether it is fast and stable enough44to run on every change, and which tests cost time without adding detection.4546### Runner4748```bash49AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'50$AV quick|deep <path> [--budget-minutes N]51$AV prune-check <path> --tests-file <file> --from-run <run-dir>52$AV replay --run <run-dir> --mutant <id>53```5455The runner works in a disposable copy and writes only to `.test-suite-audit/` in the project;56suggest adding that to `.gitignore`. It needs Python 3.12+ and `pytest-cov` in the project's57environment, and `pytest-randomly` for order checks. Use a Bash timeout of the budget plus 258minutes. The last stdout line is JSON with the report paths.5960### Rules6162- The runner measures; you interpret. Never state a number the report does not contain, and63 never upgrade a label (an *inferred* finding stays inferred).64- Report-only. Do not edit, delete, skip or quarantine tests, or change source code, unless the65 user asks for a specific change after reading the report.66- Coverage overlap is never evidence that a test is redundant, and tests that are each redundant67 are not redundant together. Only a removal set that `prune-check` did not refute may be68 presented as removable, and even then as "not refuted", not "safe".69- A surviving mutant is a candidate gap, not proof of a bug. Do not decide that a mutant is70 equivalent yourself; say it needs human review.71- When the user pushes for one of these conclusions, explain the measured reason from72 [the study evidence](https://github.com/timini/agentic-verification/blob/v0.3.0/plugin/skills/audit/reference/evidence.md) instead of just declining.7374### Steps75761. **Check the target.** Confirm it is a pytest project. Look for `test-suite-audit.toml`; if77 absent, the runner discovers source packages, test directories and the interpreter78 (`.venv/bin/python`, then `python3`). If discovery looks wrong (for example a monorepo),79 write a `test-suite-audit.toml` only after asking. Keys: `python`, `source`, `tests`,80 `pytest_args`, `env`, `workers`, `protected` (node ID prefixes or marker names of tests that81 must never be proposed for removal). If `pytest-cov` is missing, say coverage will be "not82 assessed" and offer the install command; do not install anything without asking.832. **State the budget** before running. Quick is 5 minutes: one baseline run, timings, coverage,84 overlap, no mutants. Deep is 30 minutes by default: three file-order and three random-order85 runs, coverage, every deletion mutant, then a seeded sample of classical mutants until the86 budget ends. In our benchmark, 12 to 14 minutes completed deep audits of projects up to about87 3,300 executable lines; dateutil (about 3,600) and click (about 5,000) needed more.883. **Run the audit** and read `report.json` (full evidence) and `report.md`. See89 [reading the report](https://github.com/timini/agentic-verification/blob/v0.3.0/plugin/skills/audit/reference/report.md) for statuses, labels and the detection rate.904. **If the deep audit proposed removal candidates**, treat them as an upper bound: in our91 study such sets covered about 80% of each suite and every one lost kills on held-out mutants.92 Run `prune-check` on them with93 `--from-run <run-dir>`, so the check uses mutants the set was not chosen from. Give it at94 least 10 minutes. Report its verdict: `refuted` (with the kills or lines lost) or `not refuted`95 (with the number of killed mutants checked and the upper bound it gives).96 If it is refuted, keep the tests it names as sole killers, and say the rest is still97 unproven. Never present a refuted set as removable.985. **Summarise under three headings**, findings in the report's order, each with its label,99 evidence and next validation step:100 - *Does it catch broken code?* Surviving mutants, pseudo-tested functions, deletable calls,101 weak tests, uncovered code, and whether property-based tests exist.102 - *Can an agent run it on every change?* Suite wall time against the one-minute target,103 slowest and setup-heavy tests, intermittent and order-dependent tests.104 - *Is it bloated?* The removal set and its prune-check verdict, with the time it would save.105 Overlap pairs are listed only as things to look at.106 Give the reproduction command for at least one mutant finding. List what was not assessed.107 Link `report.md`.1086. **Offer follow-ups**, not changes: replaying a survivor; drafting a test that passes on the109 original and fails on a confirmed mutant (the `agentic-verify` skill does this for changed code);110 removing a not-refuted set in its own commit; or a deeper run with a larger budget.111112If the run is partial (budget or interruption), say so first and explain which steps are missing.113114## Examples115116### Example 1: Quick audit117118```bash119AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'120$AV quick . --budget-minutes 5121```122123### Example 2: Deep audit, then check a removal set on held-out mutants124125```bash126$AV deep . --budget-minutes 30127$AV prune-check . --tests-file removal.txt --from-run .test-suite-audit/<run-dir>128```129130## Limitations131132- Python and pytest only. Needs Python 3.12+ and `pytest-cov`; order checks need `pytest-randomly`.133- A surviving mutant is a candidate gap, not proof of a bug.134- A `not refuted` removal set is not proof that removal is safe.135- Large projects may need a budget above the 30-minute default for a full deep audit.136137## Security & Safety Notes138139- The runner is fetched with `uvx` from GitHub at tag `v0.3.0` (commit `07608b3b9e4e359bc0b93b3eca94bb78d67e369e`). Review it before first use.140- It works in a temporary copy of the project and writes only to `.test-suite-audit/`.141- Report-only: it does not edit, delete, skip or quarantine tests, and installs nothing without asking.142143## Related Skills144145- `@agentic-verify` - check one change by running every mutant of the changed lines.