Bullshit Tests

Bullshit Tests

Find tests that pass for no good reason.

A test suite's value is entirely in its ability to fail when the code is wrong. A test that cannot fail is worse than no test: it costs CI time, it costs maintenance, and it buys false confidence. This skill audits a test suite for those tests and reports them.

Deterministic first, judgment last. A bundled script (scripts/ledger.py) enumerates every test and does the mechanical triage — the part a linter does better than a model — exhaustively and for free. The LLM is spent only on the judgment a script can't do: name-vs-behavior, assertion-true-by-construction, artifact-vs-behavior, and the false-positive filtering. That split is what makes this work on a suite of thousands without missing tests, inventing findings, or burning tokens reading code that a grep already cleared. The ledger tracks coverage as a number, so "we checked everything" is verifiable rather than asserted.

Model routing

Two roles, two models. The models are configurable — the defaults are below, and the user may override either. What must not change is the separation of concerns: a cheap model does the high-volume reading and hands candidates to a stronger model; only the stronger model writes a verdict.

Two constraints follow, and they hold whatever models the user picks:

1. The fan-out never writes a red/yellow. It surfaces candidates; the judgment model adjudicates them. A cheap model wrongly clearing a green is a recall miss (backstopped by the ledger's mechanical triage and a spot-check); a cheap model writing a verdict is a fabrication, which the whole skill exists to prevent.
2. The fan-out returns structured output. Its job is to feed another agent, so it must emit a machine-consumable payload against a fixed schema — not prose the primary has to re-parse. The schema is in Step 3.

The judgment model rides the session's main loop, so the skill can't set it — the user launches the session on their chosen judgment model. The skill can set the fan-out model, and does (Step 3).

Scope

This skill diagnoses. It does not fix. Do not rewrite, delete, or repair tests unless the user asks in a follow-up. The deliverable is a report.

It audits signal, not suite economy. A test that passes for no good reason is in scope. A test that is an exact duplicate of another, or pins behavior nobody wants, or covers dead code is waste — it can still fail, so it isn't false confidence. Note it in passing if you trip over it, but that's a different audit; don't stretch this one to cover it.

Hard rule — never modify the user's code

Under no circumstances modify any file in the user's working tree — above all on main or any other shared/default branch. Not source, not tests, not "just for a second, I'll revert it." Other people work in that checkout, and a clean git status afterward is not proof that nothing went wrong in between. Writing the report into the audited repo counts as touching it (Step 6).

This skill reaches its conclusions by reading, which is nearly always sufficient. When it isn't, you stop and ask (Step 5). A weaker finding always beats touching a checkout someone else is using.

The one question

For every test examined, answer exactly one question:

If I broke the code this test claims to cover, would this test go red?

If the answer is "no", it's RED. If the answer is "only for some breakages, and not the obvious ones", it's YELLOW. Everything else is GREEN.

Judge the promise-signal pair. A test is a promise — its name, plus whatever its docstring and comments claim — bound to a signal, which is whatever its assertions can actually detect. The defect is always the gap between them. Which side gets fixed (rename the test, or strengthen it) is the maintainer's call, not yours; report the gap and leave the direction to them.

Never flag a test on pattern-matching alone. Every finding must come with a concrete failure story: a specific way the production code could be broken while this test stays green. If you cannot write that sentence, you do not have a finding — drop it.

This rule buys precision by paying recall, deliberately. Crying wolf once costs you every later finding. But it under-reports diffuse rot that resists a one-sentence story — snapshot decay, expected values retrofitted to match output, suites that only ever test the happy path. Say so in the report's Coverage section.

Workflow

Step 1 — Confirm scope

First, establish the ground state and report it in the same message you ask in: current branch, whether the working tree is clean, and whether the repo supports git worktree. Never start an audit without knowing what branch you're standing on.

Then ask, in one message, only what you actually need: which directory / package / test suite to audit, if the repo has more than one, and how deep to go (full read vs. a sample, for a suite in the thousands).

Do not ask about mutation up front. The audit is read-only, and most findings never need it. If a specific finding later turns out to be unresolvable by reading, ask then — about that one finding, with the reason (Step 5).

If the user already named a path or said "the whole thing", just proceed — but still report the branch and tree state.

Step 2 — Map the suite, then build the ledger

Why a ledger. On a suite of thousands of tests you cannot read every test in one context, so reading-only means sampling, and sampling silently misses. The bundled scripts/ledger.py fixes that: it enumerates every test into a JSONL worklist in scratch space, triages the mechanically-detectable cases deterministically (free, exhaustive, no hallucination), and tracks a status per test. Coverage becomes a number you can check — pending — not a claim. The work is resumable and fannable-out: any number of turns or subagents can draw pending rows and record verdicts without losing place.

First, know the stack so you read it idiomatically — runners (pytest, Jest, Vitest, Mocha, JUnit, Go testing, RSpec…), config files, and above all the shared helpers, custom matchers, fixtures, and conftest. Read those first; most false positives come from an assertion living in assert_valid_user() or a roundTrips() helper.

Then build the ledger (writes to scratch, never the repo):

python3 <skill>/scripts/ledger.py build <testdir> --out <scratch>/audit.ledger.jsonl

It reports the inventory and the triage: high = mechanical candidates (a flag fired — these are where reds concentrate), med = thin (0–1 assertions, need a read), low = likely green (≥2 assertions, no flag).

The enumerator parses Python (pytest/unittest), JavaScript/TypeScript (Jest/Vitest/Mocha, including __tests__/ dirs), Go (testing/testify), Ruby (RSpec/Minitest/Rails), and Rust (#[test] family). For any other language it finds nothing and prints a warning — if the suite is in Kotlin, C#, PHP, Elixir, etc., the ledger is empty and you must say so, not report a clean bill of health. Even for a supported language, the count is only as complete as the regex parsed (e.g. a JS file outside the naming conventions is missed); if the total looks low against the codebase, flag it. Report the counts to the user before adjudicating.

Step 3 — Work the ledger, highest-priority first

The mechanical sweep is already done and it was exhaustive; the LLM's job is the judgment the script can't do. Pull batches and adjudicate — do not re-derive the whole suite by reading it top to bottom.

python3 ledger.py next  <ledger> --priority high        # every mechanical candidate — read all
python3 ledger.py next  <ledger> --priority med         # thin tests — read all
python3 ledger.py stats <ledger>                        # your progress bar (pending count)

• high and med: read each one (Step 4) and record a verdict. These are the whole judgment surface — usually a small fraction of the suite.
• low: do not read all of them. Read a disclosed sample (say how many), enough to validate that "≥2 assertions, no flag" really does mean green in this codebase. If the sample turns up a bullshit test the flags missed, widen the sample and note the miss.

Fanning out on a large suite (the fan-out model)

For a big suite, parallelize the reading, not the verdicts. Spawn fan-out subagents on the configured fan-out model (default Haiku-tier — pass model: haiku on the Agent call), each handed a disjoint slice (a --priority/--flag filter or a file range). Their job is to clear obvious greens and flag suspects — never to write a red or yellow. Every real finding is adjudicated by the judgment model (Step 4).

The fan-out is providing input to another agent, so it returns structured output, not prose. Instruct each subagent to reply with a JSON object against exactly this schema and nothing else:

{
  "reviewed": [
    {
      "id": "tests/foo.py:42",          // ledger id, verbatim
      "call": "green" | "suspect",       // green = confidently real; suspect = needs judgment
      "reason": "one line: why green, or what smells off",
      "evidence": "file:line the reason rests on, or null"
    }
  ],
  "unsure": ["tests/bar.py:9"]           // ids it couldn't reach a call on — treat as suspect
}

The primary consumes that payload deterministically: suspect and unsure rows go to Step 4 for adjudication on the judgment model; green rows are bulk-set green, with a disclosed random sample re-read by the judgment model to confirm the fan-out's greens hold (a wrongly-cleared green is a miss, so sample enough to trust the heuristic). A fan-out subagent must never call ledger.py set --status red/yellow; if it thinks it found one, that's a suspect.

Record every verdict with the tool — this is what makes coverage auditable and keeps you honest:

python3 ledger.py set <ledger> <id> --status green --pattern "..." --verified read
python3 ledger.py set <ledger> <id> --status red   --pattern "..." \
        --severity "only guard on X" --evidence src/foo.py:88 --story "..." --verified read

The tool refuses a red/yellow without both --evidence (a real file:line) and --story. That is the structural brake on fabrication: no accusation without its receipt.

Step 4 — Adjudicate one row by reading it

For each high/med row (and each sampled low), open the test and everything it calls. Assign a verdict only when you can write the failure story. Check:

• Does a helper, fixture, custom matcher, or teardown hook contain the real assertion?
• Is this a deliberate smoke/import/compile/type test? Those are legitimate — but they belong in YELLOW if their name promises behavior they don't check.
• Is the mock assertion actually a contract test (the point is that the collaborator gets called correctly)? That's legitimate GREEN.
• Does the framework fail the test implicitly (unmatched HTTP mocks at teardown, strict mock verification, --strict-markers, snapshot obsolescence checks)?

Cite where you verified any runtime-semantics claim. This is the failure mode that makes a wrong finding more convincing than a right one. If a failure story rests on a claim about what the code does — "parse_page raises on an empty title", "this helper returns None on miss" — you must have read the line that establishes it, and checked that fixtures, conftest, autouse setup, or module-level patching don't replace that symbol as the test actually imports it. In dynamic languages a confident static read can be confidently wrong, and the failure-story rule will dress the error up persuasively. Name the source line in the finding. If you can't verify the claim, downgrade the finding rather than assert it.

Then read the siblings — the other tests in the same file and module. This is required, not optional: it's what tells you whether the gap you found is a shipping risk or a redundant test with better coverage next to it. It also catches your own overreach; a finding that looks severe alone often turns out to be guarded three tests down.

Assign a category and a severity (both factors — see Categories). Only report findings you'd defend.

Step 5 — Blast radius (usually: don't run code)

Reading establishes whether a test can fail, and reading the siblings establishes whether the module covers the gap — both are part of Step 4's failure story. What reading can't do at acceptable cost is the suite-wide claim: nothing anywhere catches this. That claim needs execution, and it's the one thing that turns "this test is weak" into "this behavior ships unverified."

Don't improvise it. For anything systematic, recommend a real mutation tool in the report — mutmut, cosmic-ray, Stryker, PIT — pointed at the module. They do this properly, and they don't need to touch anyone's checkout.

For a one-off on a headline finding, ask first, offering: (1) skip it — report as READ with its failure story, usually the right call; (2) an isolated copy (git clone / cp -r to /tmp); or (3) an issue + branch + throwaway worktree (git worktree add /tmp/bullshit-tests-<slug> -b <branch>). Wait for a real answer — an earlier "yes, audit this" is not consent to edit their code.

If approved: inside the isolated copy only, break the code crudely (wrong constant, raise NotImplementedError), re-run the full suite — that's the whole point, a single-test re-run only re-proves what reading showed — then restore before the next finding. Still green → mark it CONFIRMED and record the exact mutation as evidence. Went red → the test was fine; drop the finding and say so.

Step 6 — Write the report

The ledger is the source of truth. Generate the skeleton from it, then write the prose:

python3 ledger.py stats  <ledger>     # final counts + the pending number
python3 ledger.py report <ledger>     # verdict table + every red/yellow with its receipt

If pending is non-zero, the report must say so — coverage is incomplete and the percentages are partial. Do not round a partially-worked ledger up to "audited." Follow references/report-template.md for the shape.

Never write the report into the audited repo. It is your output, not their source tree, and it should not show up in their git status or risk being committed.

Ask the user where they want it before writing it. Offer a sensible default so it's a one-word answer — e.g. "Where should the report go? Default is your scratchpad (<path>); or name a path (not inside the repo)." Use whatever they say.

Only skip the question when you can't ask — a named path already given earlier, or a non-interactive run. Then fall back in this order and state where it went: (1) the path they named, (2) the session scratchpad, (3) ~/scratch/ if it exists, (4) /tmp/.

Filename: bullshit-tests-report-<repo>-YYYY-MM-DD-HHMM.md, using the real current date/time — look it up, don't guess.

Then print to the user: the counts per category, the three worst findings, and the path.

Categories

Judge by signal, not by style. A short test with one sharp assertion is GREEN.

🔴 RED — cannot fail

Passes regardless of what the production code does. Zero signal, false confidence.

🟡 YELLOW — weak or misleading signal

Stays green under a breakage that a reader of the test's name would expect it to catch. (Not "misses some breakage" — every test misses some breakage.)

🟢 GREEN — real signal

Asserts observable behavior of the code under test, using values it did not itself supply, and would go red if that behavior changed. Includes legitimate contract tests on mocks, deliberate and clearly-named smoke tests, and property-based tests.

Severity is two-factor

The category above is only factor one. Before assigning severity, answer factor two: is the behavior this test names covered by anything nearby? Read the sibling tests in the same file and module — that check is part of the failure story, not optional.

• Weak test, strong sibling coverage → a cleanup note. Real, but nobody needs to act.
• Weak test that is the only guard on a shipped behavior → this is the finding the whole report exists for. Say so explicitly and rank it first.

A list of individually-correct verdicts that doesn't separate these two is worse than useless — it reads as uniform risk and buries the one thing worth doing. The suite-wide version of factor two ("nothing anywhere covers this") can't be established by reading; see Step 5.

Rules

• No fixes. Report only, unless asked.
• No pattern-only findings. Failure story or it doesn't ship.
• Read helpers before judging. The assertion is often one call away.
• Rank by factor two. The only-guard-on-shipped-behavior findings go first. Everything else is a cleanup note, and should read like one.
• Don't moralize. Report the pattern, the risk, and the file:line. Some of these were deliberate trade-offs by people with more context than you.
• Say what you didn't cover — sampling, skipped directories, and the recall blind spot above. A silent cap reads as full coverage.
• Announce every suite run before it happens, with why and roughly how long. Test runs are minutes of the user's time; don't spend them silently.
• Never edit the user's tree; never write the report into their repo. See the hard rule.