Auto Test

Auto Test

Overview

Drive a codebase from "some/unknown test health" to "a green suite that meaningfully covers the target", autonomously and safely. The value is the combination: it loops (find gaps → write → run → fix → re-run) but it stops honestly (bounded, fails closed) and it doesn't cheat (every test is proven able to fail, the suite is never gamed green). It runs the converge-loop until-green pattern, gates each new test through adversarial-verify (mutation check), and checkpoints via checkpoint-resume.

When to Use

• After auto-build / any implementation, to cover the behavior that was just written
• To raise real coverage on a specific risky module (auth, money, data migrations, parsers)
• To add a regression test that reproduces a reported bug before it's fixed (the Prove-It pattern)
• To de-flake an unreliable suite (auto-test flaky)

When NOT to use: writing tests test-first while implementing new behavior (that's auto-build's RED→GREEN loop); a repo with no runner configured (set one up first); pure-config/docs changes with no behavioral surface.

Phase 0: Ground the run — scope, runner, baseline, methodology

Before writing anything:

1. Resolve scope ($scope): a path/module, the current diff (default — git diff against the base), or flaky (stabilize mode). Narrow, targeted scope beats "test the whole repo."
2. Detect the runner + coverage tool — read the repo: package.json scripts, pytest.ini, Cargo.toml, a Makefile. Capture the exact command to run the whole suite and to run a single file/test (you'll use the single-test form for tight loops). If none exists, STOP and say so.
3. Baseline the signal — run the suite ONCE. Record: pass/fail counts, which tests fail, and (if available) current coverage on the scope. This is converge-loop iteration 0. If it's already green with the target covered, there may be nothing to do — say so, don't invent busywork.
4. Load our test-quality bar — this skill carries its own standard for what a good test is (see references/test-quality-bar.md): the test pyramid (≈80/15/5), test sizes, DAMP-over-DRY, state-not-interaction assertions, real-implementations-over-mocks, and the anti-patterns to avoid. Apply it directly; it is self-contained, not a pointer to another collection.
5. Open a checkpoint — start a checkpoint-resume run (.ulpi/runs/<id>.json) with one unit per target behavior/file. On resume, skip units already done.

Success criteria: scope fixed; exact suite + single-test commands known; a concrete baseline (counts + failing tests + coverage) recorded; checkpoint open.

Phase 1: Find the gaps — the work list

Identify the behaviors that lack a meaningful test (not just uncovered lines):

• diff the scope against the tests that touch it; list public functions/branches/error paths with no assertion behind them;
• prioritize by risk — untested error/edge paths, money/auth/data-mutation code, and recently changed code rank above cosmetic getters;
• for flaky mode, instead identify the tests that fail intermittently (run the suite N times, collect the non-deterministic failures) — those are the units.

For a large scope, fan the discovery out with fan-out-work (one agent per module) and merge the gap list. Record each gap as a checkpoint unit.

Success criteria: a prioritized, de-duplicated list of concrete missing tests (or flaky tests), each an addressable unit.

Phase 2: Write one meaningful test — and prove it can fail

Per unit (smallest first), write ONE focused test, then verify it's real BEFORE trusting it:

1. Write it following the methodology — Arrange-Act-Assert, a descriptive name that reads like a spec, one concept per test, state-based assertions, real implementations over mocks. For a bug repro, write it to FAIL against current code (RED).
2. Mutation-check it (adversarial-verify for tests): make a small, targeted break in the code under test (flip a comparison, drop a write, return a wrong constant) and re-run just this test — it MUST go red. Restore the code — it MUST go green. A test that stays green on the broken code is a tautology: reject it, rewrite it to actually assert the behavior. For heavy verification, delegate the mutation-probe to a subagent.
3. Classify a genuine failure — if, against the correct code, the test fails, decide: is the TEST wrong (fix the test) or is the CODE wrong (a real bug — surface it; fix only if in scope, never by asserting the buggy output)?

Success criteria: each added test provably fails when its target is broken and passes when it's correct; any real bug the test exposed is recorded, not papered over.

Phase 3: Converge the suite to green

Run the converge-loop until-green pattern over the whole target, with its full termination set (done = suite exits 0 over the scope; maxIterations; token budget; maxStall=2):

• after each added/fixed test, run the suite (single-file form for speed during the loop; full suite at round boundaries to catch regressions);
• if a change regresses another test, revert and reconsider — never ratchet the suite backwards;
• if a unit can't be made green in MAX_FIX (≈3) attempts, mark it blocked with the reason and move on — do not spin;
• update the checkpoint as each unit reaches done / blocked.

Success criteria: the scoped suite is green (or the loop terminated honestly with the specific blocked units named); no regressions introduced.

Phase 4: De-flake (stabilize mode, or any flake surfaced)

For any test that passes/fails non-deterministically:

• reproduce by running it in a loop / with randomized order; find the root cause class — shared state, time/timezone, ordering, real network, unawaited async;
• fix the ROOT (isolate state, inject the clock, await properly, fake the boundary) — never "fix" a flake by adding a retry or a sleep to mask it;
• confirm stability: N consecutive green runs (and randomized order) before calling it fixed.

Success criteria: previously-flaky tests pass deterministically across repeated + reordered runs; no flake masked by retries/sleeps.

Phase 5: Report

Finalize the checkpoint and report honestly (see Output Contract). Include the before→after signal (counts + coverage delta), the tests added, any real bugs surfaced, and any blocked units.

Common Rationalizations

Red Flags

• The suite went green in the same edit that deleted/skipped/.only'd a test.
• A newly added test passes against a deliberately broken version of the code.
• An assertion was changed to match the code's current (possibly wrong) output.
• A flake "fixed" by a sleep, a retry wrapper, or an increased timeout.
• Coverage % climbing while assertions are vacuous (toBeDefined, not.toThrow on everything).
• "All tests pass" reported without a suite run in the transcript.
• The loop is on its 6th iteration re-trying the same failing approach (thrash — stop and escalate).

Enforcement (deterministic, not prose)

While this skill is active, a skill-scoped PreToolUse hook runs scripts/guard-test-integrity.sh on every Edit/Write: adding a .only/.skip/xit/xdescribe/.todo/@pytest.mark.skip/@unittest.skip/ #[ignore] marker or a @ts-ignore/@ts-expect-error/eslint-disable/# type: ignore suppression to a test file is BLOCKED at the tool layer. That stops the skip/only/ignore/suppression class of gaming at the Edit/Write layer (a raw-Bash write of the same marker sidesteps this Edit/Write-scoped hook — caught, like the vectors below, by mutation-check discipline). The OTHER cheat vectors this skill forbids (deleting a test, a vacuous expect(true), weakening an assertion, masking a flake with sleeps) are not statically detectable at the edit layer — they stay enforced by the mutation-check discipline and the fail-closed contract above, not by this hook. Genuine, user-approved weakening goes through the explicit escape hatch — touch <project>/.ulpi/allow-test-weaken opens a 2-minute approval window (then expires; AUTO_TEST_ALLOW_WEAKEN=1 exists for settings-level use) — with the reason stated in the reply.

Guardrails

• Never weaken, skip, delete, or .only tests to reach green. Fail closed instead.
• Never count a test that can't fail — mutation-check every addition.
• Never rewrite a test to assert buggy behavior; surface the bug.
• Never mask a flake with sleeps/retries/timeouts; fix the root cause.
• Never chase a coverage number with vacuous tests.
• Never report green without a final real suite run and its exit code.
• Keep each iteration small and measured (one test/behavior); revert any regression immediately.
• Escalate ambiguous expected-behavior questions instead of guessing.

When To Load References

• converge-loop (skill) — the until-green loop with the termination set + anti-thrash. The engine of Phase 3.
• adversarial-verify (skill) — the mutation-check / tautology-rejection gate for Phase 2.
• checkpoint-resume (skill) — the durable run state for skip-done resume.
• fan-out-work (skill) — parallel gap discovery / test writing over a large scope.
• references/test-quality-bar.md — OUR standard for a good test: pyramid, sizes, DAMP, state-not-interaction, real-over-mocks, and the anti-patterns. Load in Phase 0.

Verification

Before reporting done, confirm:

• The scoped suite passes on a fresh, real run (exit code read, not assumed)
• Every test added this run was mutation-checked (fails on broken code, passes on correct code)
• No test was skipped, deleted, .only'd, or weakened to reach green
• Any real bug a test exposed is surfaced (and fixed only if in scope — never asserted-as-correct)
• Flaky tests (if any) pass deterministically across repeated + reordered runs, with no masking
• Coverage delta (if tracked) reflects real behaviors, not vacuous assertions
• The checkpoint file reflects the final per-unit state; blocked units are named with reasons

Output Contract

Report:

1. scope + suite command used; baseline → final signal (pass/fail counts; coverage delta when the repo tracks coverage)
2. tests added (by behavior), each noted mutation-verified
3. real bugs surfaced (and whether fixed in scope or handed off)
4. flakes stabilized (root cause + how) — if any
5. loop outcome: converged, or the honest list of blocked/failing units with reasons
6. checkpoint file path (durable record; resume-able)