Testing Boss

Testing Boss

A consolidated doctrine for writing tests that reveal bugs, not just pass — for human-authored code, AI-generated code, LLM-powered features, and the CI that gates them all.

The cardinal premise: tests exist to expose defects, not to keep CI green. A test that fails has done its job. A test that passes for the wrong reason is worse than no test.

This skill collapses the old test-antipatterns skill plus a much larger corpus on test placement, framework idioms, flaky-test discipline, AI-agent test generation, and LLM/agent evaluation into one self-contained body of practice. Examples are language-agnostic pseudo-code so the doctrine transfers to any stack.

Iron Laws

1. Test the behavior, never the mock.
2. Push every test to the lowest layer that can detect the failure.
3. When a test fails, fix production first — change the test only after writing why.
4. Real systems gate the merge. Mocks isolate; they do not validate.
5. Coverage is a flashlight. Mutation score is a quality probe. Neither is a target.
6. No test-only methods, branches, or flags leak into production code.

These six laws subsume every named anti-pattern in this skill. When two of them disagree, the lower-numbered one wins.

Required Reading Router

Match the task to the row. Read the listed file(s) in full before producing output. The inline content in this SKILL.md is a tripwire, not the contract.

Reference Index

• references/foundations.md — placement doctrine (invariant + owning layer), pyramid vs trophy debate resolution, risk-based prioritization, coverage philosophy, test-boundary contracts.
• references/patterns.md — 12 cross-framework positive patterns with agnostic pseudo-code: selector hierarchy, condition-based waits, per-test isolation, table-driven, builders/factories, behavior-first assertions, boundary-only mocking.
• references/antipatterns.md — 25 anti-patterns across five families (Brittleness, Flakiness, Mock misuse, Process, AI-specific). Each entry: violation, why wrong, fix, gate question, evidence URL.
• references/ai-writes-tests.md — seven mandatory gates with verbatim prompt blocks for any agent that generates tests: invariant first, owning layer, real execution, failure→fix production, no snapshot without contract, no assertion on self-set mock, negative companion.
• references/ci-automation.md — flaky-test taxonomy, quarantine-plus-owner workflow, CI stage pyramid, contract / property / mutation / accessibility testing patterns, deterministic test architecture.
• references/llm-eval.md — eval-driven development primer, oracle ladder, LLM-as-judge biases and calibration, RAG metrics, agent trajectory vs outcome eval, benchmark pitfalls.
• references/sources.md — consolidated bibliography (all URLs grouped by axis) for citation and audit.

Decide before the first line of test code

Most bad tests are placement failures, not assertion failures. A test in the wrong layer is brittle, slow, and duplicates work — or worse, it locks in implementation under the disguise of correctness.

Gist tripwires:

• Name the invariant in one sentence before opening any test file. If the sentence is fuzzy, the invariant is not clear enough to test.
• Place the test at the lowest layer that can fail when the invariant breaks. A higher-layer test is justified only when the invariant requires real integration the lower layer cannot prove.
• Reject the test entirely when (likelihood-of-bug × blast-radius) is below the threshold worth ten minutes of maintenance. Not every line deserves a test.

STOP. Read references/foundations.md in full before placing a new test, splitting a test across layers, debating pyramid vs trophy, or arguing about coverage targets. That file contains the placement decision tree, the explicit pyramid/trophy reconciliation, the test-boundary contract template, and the risk-based filter. The three tripwires above are detection cues, not the contract.

Pattern catalog (write tests that survive refactor)

Twelve patterns recur across Playwright, Testing Library, Cypress, Jest, pytest, Go testing, and Pact. The framework is evidence; the principle is universal.

Named patterns (one-liners — full pseudo-code in the reference):

1. Query by behavior and accessible role, never by CSS selector or DOM index.
2. Selector hierarchy: role → label → text → test-id → structural. Stop at the highest rung that disambiguates.
3. Wait on observable conditions, never on wall-clock sleeps.
4. Each test is independent and order-free; setup beats teardown.
5. One behavior per test, but as many assertions as that behavior needs.
6. Test names read as specifications: should <outcome> when <condition> given <state>.
7. Table-driven / parameterized when only the inputs vary.
8. Build test data via factories or builders; literal blobs only for the field under test.
9. Mock at boundaries you do not control; real wiring for what you own.
10. Real systems gate the final merge; contract tests bridge unit and E2E.
11. Mutation score, not coverage percentage, measures suite strength.
12. Page Object Model is a tool, not a religion — collapse it for small suites.

STOP. Read references/patterns.md in full before writing any non-trivial test, choosing a selector strategy, designing test data, or deciding what to mock. That file contains the pseudo-code, the cross-framework evidence, and the explicit "when to break this rule" carve-out for each pattern. The twelve one-liners above are a vocabulary index, not the contract — the operational rule for each pattern lives only in the reference.

Anti-pattern families (do not do these)

Twenty-five anti-patterns cluster into five families. The top seven (bolded below) cause the most damage in modern codebases — especially when AI agents write the tests.

Brittleness — tests bound to internals.

1. Brittle/implementation-detail selectors.
2. Testing internal structure instead of observable behavior.
3. Testing private methods directly.
4. Snapshot-as-test (a snapshot replacing real assertions).
5. Vague existence assertions (.should('exist'), toBeTruthy).
6. Action without assertion.

Flakiness — tests that randomize their own verdicts. 7. Static sleep / fixed-timeout waits. 8. Test order dependency / hidden shared state. 9. Non-deterministic inputs (real clock, RNG, locale).

Mock misuse — tests that test the test setup. 10. Asserting the mock exists. (absorbed from the previous test-antipatterns skill) 11. Mock drift (mock no longer matches real API). 12. Over-mocking child components. 13. Incomplete mocks (missing fields the system consumes downstream). 14. Mocking at the wrong level (mocks slow-and-safe methods test logic depends on).

Process — pathologies of the team and the suite over time. 15. Coverage as a vanity metric. 16. Happy-path-only coverage. 17. Eternal beforeAll / shared setup that hides dependencies. 18. Cleanup in afterEach (use beforeEach instead). 19. Magic strings and logic in tests. 20. Testing against third-party sites you do not control. 21. Quarantine-as-cemetery (skip without owner or deadline). 22. Retry-as-fix (auto-retry hiding real bugs). 23. Duplicate tests across pyramid layers. 24. Weakening tests to make them pass. 25. Mock-driven confidence (the test sets up a mock and then asserts on its own setup).

The old test-antipatterns skill covered five of these (Asserting the mock exists, Test-only methods in production, Mocking without understanding, Incomplete mocks, Integration tests as afterthought). All five survive here — folded into Mock misuse and Process — and the framing question "Are we testing the behavior of a mock?" still applies.

STOP. Read references/antipatterns.md in full when reviewing any test, debugging a flaky suite, refactoring tests after a refactor broke them, evaluating AI-generated tests, or deciding whether to delete or rescue a struggling test file. That file has the full 25-entry catalog with violation pattern, why-wrong, fix, gate question, and citation per entry. The family list above is a topic index, not the contract — flagging a pattern by family is not the same as knowing the fix.

AI agents writing tests

Coding agents will mock everything to green by default. They produce long, linear tests with high assertion density and no edge cases — and they will happily patch the test instead of fixing the code. The skill ships seven gates to block that.

Gate names (the prompt blocks live in the reference):

1. Invariant first — agent prints INVARIANT: …, OWNING_LAYER: …, EXISTING_SUITE: … before any test code.
2. Owning layer — extend an existing suite; reject new files without a named invariant.
3. Real execution — every new test must run against a real DB / route / external integration at least once.
4. Failure → fix production — on red, the next tool call reads the production code, not the test.
5. No snapshot without contract — classify the artifact as PRODUCT_CONTRACT or IMPLEMENTATION_DETAIL; the latter forbids snapshots.
6. No assertion on self-set mock — cannot assert on a value the same test body wrote into a mock.
7. Negative companion — every positive assertion ships with a negative test for invalid input or failure mode.

STOP. Read references/ai-writes-tests.md in full before letting any agent generate, modify, or "fix" tests in this repository. That file contains the verbatim prompt blocks to paste into CLAUDE.md, the failure-protocol template, the mock-budget rule, and the evidence base (Anthropic's eval guide, the Yoshimoto study on AI test smells, the Stanford vibe-coding vulnerability finding). The seven gate names above are not enforceable on their own — the agent-runnable prompt block lives only in the reference.

CI & flaky discipline

Tests that pass intermittently are worse than tests that always fail. They train the team to ignore signal.

Gist tripwires:

• Quarantine a flaky test the same hour it is detected. Assign a named human owner within 24 hours with a fix-by date. No anonymous quarantines.
• Track flaky_rate as a first-class operational metric. SLO < 1–2%; alert at > 5%. Retry without telemetry is debt accrual, not stability.
• Real systems at the final gate. Mock at unit; contract-test the boundary; real DB / queue / route at integration; near-zero mocks at E2E.

STOP. Read references/ci-automation.md in full before designing CI stages, introducing retries, choosing between integration and contract tests, adding property or mutation testing, or building a regression pack. That file contains the flaky taxonomy (Async 45%, Concurrency 20%, Order 12%, …), the quarantine workflow, the CI stage pyramid, and the contract/property/mutation/accessibility patterns. The three tripwires above are detection cues, not the contract.

LLM and agent evaluation (Part 6, enxuta)

Testing an LLM feature is conventional testing with one twist: the oracle is probabilistic. Everything else — invariants, regression, traceability, real-system validation, repair-production — applies harder, not less.

Gist tripwires:

• Start small. Twenty unambiguous tasks drawn from real failures beat two hundred synthetic ones.
• Climb the oracle ladder: exact / schema / outcome-state checks before LLM-as-judge before human review. Use the cheapest oracle that catches the failure.
• LLM-as-judge needs calibration. Validate against humans (target ≥ 0.80 Spearman) before trusting any judge in CI. Always use a different model than the system under test.
• Agents need outcome checks. Trajectory grading punishes valid creativity; outcome-only grading misses ghost actions where the transcript claims success and nothing changed.

STOP. Read references/llm-eval.md in full before designing an eval suite, introducing LLM-as-judge into a pipeline, debating SWE-bench / τ-bench numbers, building RAG faithfulness checks, or rolling out trace-based observability. That file contains the oracle ladder in full, the LLM-as-judge bias list, the RAG metric decomposition, agent trajectory-vs-outcome guidance, and the benchmark-pitfalls evidence (Anthropic eval guide, the 2507.02825 benchmark-validity paper, Galileo and Braintrust frameworks). The four tripwires above are detection cues, not the contract.

Red flags (cross-cutting)

These signals should always trigger "stop and think" — no matter the layer or framework.

• Mock setup is larger than the test logic.
• Test breaks when an internal method is renamed (not a public contract).
• Removing the assertion body leaves the test still green.
• Test fails when run with .only in isolation.
• sleep, Thread.sleep, or cy.wait(<number>) appears anywhere.
• Selector contains a CSS class, an index, or xpath.
• Test asserts a third-party site is reachable.
• Snapshot diffs are accepted in code review without reading them.
• Coverage percentage is the only quoted quality metric.
• Failing tests are auto-retried until green; nobody investigates.
• Skipped or quarantined tests have no named owner and no fix-by date.
• Test depends on new Date(), Math.random(), or system locale.
• afterEach resets the database (move it to beforeEach).
• An AI-written test has six+ assertions and zero edge cases.
• The phrase "I'll mock this to be safe" appears anywhere in the diff.

When NOT to use this skill

• General code review unrelated to tests — use a code-review skill.
• Library-specific debugging where the test is just the reproduction — use the library's own debugging skill.
• Non-testing CI pipeline design (deploys, artifact promotion, secrets management).
• Production observability and alerting design — those are runtime concerns, not test concerns.
• Single-line typo fixes in existing tests — the doctrine is for non-trivial work.

Bottom line

A test that cannot fail is decorative. A test that fails for the wrong
reason is misleading. Build tests that fail for exactly one reason —
the reason the invariant was violated — and trust them when they do.

Mocks isolate. Real systems validate. Coverage shines a light. Mutation
score grades the suite. Agents will reach for the mock and the snapshot;
the gates here make them put both down.

Tests reveal bugs, not just pass.