jacob-balslev Concept of the skill
What it is: Testing strategy is the discipline of choosing which behavior needs verification, which test level can observe it honestly, and what evidence proves the change is protected.
Mental model: A strategy decision has six parts: the behavior or regression target, the risk of failure, the boundary where the failure would appear, the cheapest test level that still observes that boundary, the expected failure signal, and the future maintenance cost of keeping the test.
Why it exists: Test suites fail when they optimize for visible effort instead of useful signal. They add tests that never catch regressions, mock the only boundary that could fail, or expand end-to-end coverage until feedback becomes slow and flaky. This skill keeps verification effort tied to production risk.
What it is NOT: It is not debugging an active failing test, implementing integration-test fixtures, designing contract-test provider states, writing contributor documentation, performance-load testing, or architecture planning without a concrete verification target.
Adjacent concepts: Test coverage strategy, integration test design, contract testing, end-to-end test design, debugging, refactor verification, performance engineering, and code review.
One-line analogy: A test strategy is placing sensors in a system: each sensor should sit where a meaningful failure can be observed with the least noise and maintenance cost.
Common misconception: Coverage percentage or the edited file type does not decide the right test. The behavior that could regress decides the target, and the cheapest honest observation point decides the level.
Testing Strategy
Coverage
- Test scope: deciding what behavior actually needs a test, and what does not earn the maintenance cost
- Test level selection: choosing between unit, integration, contract, and end-to-end tests based on risk and coupling
- Effort-to-risk matching: investing verification effort where regressions are most likely and most damaging
- Regression targeting: writing tests that pin the specific behavior a change risks breaking, not generic coverage
- Evidence quality: preferring concrete, reproducible verification over assumed or manual checks
- Failure-case coverage: ensuring boundary conditions and error paths are tested, not only the happy path
Philosophy of the skill
Most test suites fail the effort-to-risk test: they exercise code that will never break and skip code that breaks in production. The correct target is the behavior that ships to users, not the code you happen to have written last. Coverage percentage is a proxy, and every proxy eventually gets gamed — the real signal is regressions caught before release. A test that never fails is noise; a test that fails without isolating the cause is worse than no test at all because it wastes the next engineer's time.
Test-Level Selection
Pick the test level by the risk of the change and the coupling of the behavior, not by the file you happen to be editing. Unit tests are cheap to write and cheap to pass; integration and contract tests are where real production bugs are actually caught.
| Situation | Test level | Why |
|---|---|---|
| Pure function, single-owner, no I/O | Unit | Fast, deterministic, zero setup. If you cannot unit-test it, the function is doing too much |
| Logic that composes multiple units inside one service | Integration (in-process) | Unit tests of each piece will miss composition bugs; integration test catches real wiring |
| Behavior that crosses a service / process / network boundary | Contract | Both sides need a shared verifiable agreement; a unit test on either side misses the real failure mode |
| User-visible flow end-to-end | E2E (one or two per critical path) | Proves the full path works at least once; too expensive to run for every code path |
| Bug fix for a bug that reached production | Regression at the level where the bug slipped through | If it slipped past unit tests, a unit test won't catch it next time; write the test at the level the bug exposed |
| Behavior that is "obviously correct," unchanged for a year, no external pressure | No new test | The test would never fail; it would only add maintenance cost. Every test is a liability until it catches a bug |
Level-selection anti-patterns
- Unit testing what should be an integration test — mocking the only thing that could actually break. Fix: test the real integration, or admit the unit test proves nothing.
- Integration testing what should be a unit test — slow setup for a function that has no dependencies. Fix: extract the pure logic and unit-test it.
- E2E-testing every code path — fragile, slow, flaky. Fix: one E2E per critical user journey, unit/integration for the rest.
- Adding a test because coverage dropped — test has no regression target and never fails meaningfully. Fix: either find a real regression to pin, or delete the uncovered code if it has no value.
Evals
This skill ships local eval artifacts at skills/skills/quality-assurance/testing-strategy/evals/comprehension.json and skills/skills/quality-assurance/testing-strategy/evals/application.json. The Verification checklist below is the authoring gate for a completed test plan; the eval files are audit-loop evidence for whether the concept can be understood and applied. Do not conflate them — the checklist is for the test author, the evals are for the grader.
Verification
- The test type matches the change risk
- A behavior or regression target is explicit
- Verification evidence is concrete, not assumed
- Failure cases and boundaries are covered, not only the happy path
Do NOT Use When
| Use instead | When |
|---|---|
documentation |
The task structures explanation for a reader, not verification for a change |
debugging |
The task is chasing a known failure — strategy is planned before the failure, not after |
refactor |
The task is restructuring code; any test work is to preserve existing behavior, which belongs to the refactor skill's verification step |