Testing Strategy

Testing Strategy

A test suite has one job: tell you, quickly and truthfully, whether the system still does what it is supposed to do. Judge every testing decision against that. Coverage percentage, test count and framework choice are proxies, and proxies get gamed.

The pyramid, and why it is that shape

        ╱──────────╲        E2E              few, slow, high confidence
       ╱   5-10%    ╲                        the critical journeys only
      ╱──────────────╲
     ╱   Integration  ╲     20-30%           real database, real HTTP layer
    ╱──────────────────╲                     where most real bugs live
   ╱────────────────────╲
  ╱         Unit         ╲  60-70%           milliseconds, no I/O
 ╱────────────────────────╲                  business rules and edge cases

The shape follows from cost. A unit test runs in a millisecond and fails precisely; an E2E test takes 30 seconds, is flakier, and tells you only that something broke. You want as much signal as possible from the cheap layer, and just enough from the expensive one to know the pieces are actually connected.

Both inversions are failures. An ice-cream cone (mostly E2E) is slow and flaky, and people stop running it. A suite with no top has never tested that the parts work together — which is where integration bugs live by definition.

What belongs at each layer

Unit — business rules, calculations, state machines, validation, edge cases, error paths, pure transformations. No database, no network, no filesystem, no clock. If it needs a mock of something you own, that is usually a design signal: the logic wants to be extracted from its dependencies.

Integration — the things unit tests structurally cannot catch: does the query return what you think, does the migration apply, does the authorization middleware actually reject, does the transaction roll back, does the serialiser produce the right shape. Use a real database (a container, or a per-worker schema). A mocked database tests your mock.

E2E — the handful of journeys where failure is unacceptable: signup, login, the core action, payment, account deletion. Through a real browser, against a real stack. Five to fifteen of these, not two hundred.

Contract — between services or against third parties. Consumer-driven contract tests catch the "we changed the API and did not tell you" class of break.

Coverage — the honest version

Coverage tells you what is not tested. It tells you nothing about whether what is covered is tested well. A line executed with no assertion is covered and worthless.

// 100% coverage. Zero value.
it("works", () => { calculateTax(100, "DE"); });

A policy that works:

• No global percentage target — it produces tests written to hit lines.
• Branch coverage on business logic modules: high, and enforced.
• 100% on the money path, the authorization path, and anything computing a permission or a price.
• Coverage must not decrease on a pull request. This is the useful gate — it prevents erosion without incentivising theatre.
• Track coverage of changed lines, not the whole repository. It is the number a reviewer can act on.

Mutation testing is the real measure. It changes your code (flips a comparison, removes a line) and checks whether a test fails. A surviving mutant is a line your tests execute but do not verify. Run it on the critical modules — it is slow, so it is not a whole-suite tool, but on a pricing engine or a permission checker it is worth more than any coverage number.

Writing a good test

it("refuses a refund larger than the amount captured", async () => {
  // Arrange — explicit, local, obviously correct
  const payment = await createPayment({ captured: 5_000, currency: "USD" });

  // Act
  const result = refund(payment, { amount: 7_500 });

  // Assert — one behaviour, named precisely
  expect(result).toEqual({
    ok: false,
    code: "refund.exceeds_captured",
    maxRefundable: 5_000,
  });
});

Rules

1. Name the behaviour, not the function. refuses a refund larger than the amount captured beats test refund 2. The name is what you read when it fails at 3am.
2. One behaviour per test. Multiple assertions about the same behaviour are fine; testing three behaviours in one test means one failure hides two.
3. Arrange visibly. A reader should not have to open three fixture files to know what the input was. Prefer explicit builders over shared global fixtures.
4. Assert on outcomes, not implementation. Asserting that a private method was called locks the test to the current design and breaks on every refactor.
5. No conditionals or loops in tests. A test with an if has a branch that might never run. Table-driven parameterised tests, not for loops with logic.
6. Deterministic. Inject the clock, seed the randomness, fix the timezone, never depend on execution order or on another test's leftovers.
7. Failure message must locate the bug. expected 5000, got 7500 is useful; expected true, got false is not.

Test the error paths. Most production incidents are in code paths that were never exercised: the timeout, the malformed input, the concurrent update, the third party returning a 500. Those need tests more than the happy path does.

Mocking — the discipline

Mock at the system boundary, not inside your own design.

Heavy mocking of your own code produces tests that pass while the system is broken, because the mocks encode assumptions the real collaborator does not honour. When you find yourself mocking four of your own classes to test a fifth, the design is telling you something.

For third-party HTTP, prefer recorded fixtures or a local mock server over hand-written stubs — hand-written stubs drift from the real API silently, and you find out in production.

Never let CI call a live third party. Slow, flaky, and occasionally expensive.

Test data

• Builders with sensible defaults, overriding only what the test cares about: aUser({ role: "admin" }). This keeps the test's intent visible.
• Isolation per test. A transaction rolled back, a truncate between tests, or a schema per worker. Shared mutable state produces order-dependent failures, which are the worst kind to debug.
• No production data in tests, ever. It is a privacy violation and it makes tests non-reproducible.
• Seed randomness and log the seed, so a random failure is reproducible.
• Freeze time. Tests that depend on the real clock fail at midnight, on the last day of the month, and during DST transitions.

Flaky tests — treat as outages

A suite people re-run until green provides no information. One tolerated flake teaches the team that red means "try again", and then a real regression is merged past.

1. Detect     — track pass rate per test in CI
2. Quarantine — move it out of the blocking set immediately, with an owner and a date
3. Diagnose   — the usual causes, in order of frequency:
                 · a race between the assertion and an async operation
                   (fix: wait for the condition, never sleep for a duration)
                 · shared state leaking between tests
                 · real time, real network, real randomness
                 · test-order dependence (verify by shuffling)
4. Fix or delete — a quarantined test with no owner after two weeks is deleted.
                   An unowned flake is worse than no test

Never add blanket automatic retries. Retrying until green destroys the exact signal the suite exists to produce.

The CI gate

Every PR, blocking:   lint · typecheck · unit · integration · build · dep audit
Before deploy:        E2E on the critical journeys, against staging
Nightly:              full E2E matrix, load test, mutation on critical modules

Requirements: under ~10 minutes for the PR gate, not flaky, and not bypassable. A gate with a culture of "just merge it, CI is being weird" is not a gate.

Starting from zero

A codebase with no tests, in priority order:

1. A test for the bug you are fixing right now. Every bug fix starts with a failing test. This alone stops the codebase repeating itself.
2. E2E on the single most critical journey. One test that proves the product fundamentally works.
3. Integration tests on the money and authorization paths. Where a bug is most expensive.
4. Unit tests on the business rules as you touch them.
5. Then breadth.

Do not attempt a coverage sprint on a legacy codebase. Characterisation tests written against existing behaviour lock in the bugs along with the features. Test what you change, and let coverage grow where the work is.

Review checklist

• The pyramid is roughly the right shape
• Integration tests run against a real database
• E2E covers signup, login, the core action, payment, deletion
• Every bug fix has a regression test that fails without the fix
• Authorization is tested as a matrix (see security-hardening)
• Error paths and edge cases tested, not just happy paths
• No live third-party calls in CI
• Tests are deterministic: time frozen, randomness seeded, order-independent
• Flake rate tracked; flakes quarantined with owners
• CI blocks merge and runs in under ~10 minutes
• Coverage does not decrease; critical modules at 100% branch coverage
• Test names describe behaviour

References

• references/test-patterns.md — patterns and anti-patterns per layer, with examples
• references/specialised-testing.md — load, security, accessibility, contract, visual, chaos