Testing

Then follow the router through Stages 2-5. Most paths terminate before test doubles are ever considered. If you reach for a fake or stub without completing the router, you are doing it wrong and will be caught by the reviewing skill.

DO NOT EVEN THINK ABOUT SKIPPING ANY STEPS OR USING MOCKS, FAKES, OR ANY OTHER TEST DOUBLES WITHOUT EXPLICIT APPROVAL FROM THE USER!

No mocking. Ever. Mocking (using frameworks to intercept and stub arbitrary methods) gives you tests that pass while production code fails. This is worse than no test at all. If you feel you need to mock, either redesign with dependency injection or test at a different level.

Reality is the oracle. Test against real systems whenever possible. A test that passes against a fake, which is nothing bot your imagination of how a database works, proves nothing about the code that you could not have anticipated.

Test doubles are exceptions, not defaults. The seven exception cases in Stage 5 are the ONLY legitimate uses for test doubles. If your situation doesn't match an exception, don't use doubles.

1. What evidence you need (Stage 1)
2. At what level that evidence lives (Stage 2)
3. What kind of code this is (Stage 3)
4. Whether the real system can produce the behavior (Stage 4)
5. Which exception applies, if any (Stage 5)

Start at Stage 1. Do not skip ahead.

1. What behavior could be wrong in production? Not "what code am I testing" but what could actually fail for users?

2. If this test passes, what does that prove about the real system? A test that proves nothing about production is waste.

3. What failure would this test catch that would otherwise reach users? If you can't name a concrete failure, you don't need the test.

The Evidence Trap

Agents often skip this stage. They see code and think "I need to test this." That's backwards.

• Wrong approach: See OrderProcessor that calls repository.save() → think "I need to test OrderProcessor" → create InMemoryRepository fake → write test that passes
• Right approach: Ask "What evidence do I need?" → answer "Evidence that orders persist correctly" → realize a fake repository proves nothing about persistence → test at Level 2 with real database

Factor 1: What Does the Spec Promise?

The acceptance criteria drive the minimum test level:

Factor 2: What Dependencies Are Involved?

Factor 3: How Complex Is YOUR Code?

Trust the library: Don't test that argparse parses flags. Test YOUR behavior that uses the parsed result.

Factor 4: Debuggability Needs

When this Level 2/3 test fails, will a Level 1 test help find the bug faster?

Factor 5: Where Does Achievable Confidence Live?

Some guarantees can ONLY be proven at certain levels:

Level Selection Decision

Based on the five factors:

• Evidence lives at Level 3 → Use real environment. DONE. (Consider: add Level 1 for debuggability of complex parts?)
• Evidence lives at Level 2 → Use real dependencies. DONE. (Consider: add Level 1 for debuggability of complex parts?)
• Evidence lives at Level 1 → Go to Stage 3

If evidence requires Level 2 or 3, stop here. Use real dependencies. Do not fake what you can test for real.

3A: Pure Computation

Given inputs, compute outputs. No external state, no side effects.

Examples: validation logic, parsing functions, business calculations, data transformations, command/argument building.

Decision: Test directly. No doubles needed. DONE.

3B: Code with Dependencies - Can You Extract?

Can you extract the pure computation from the dependency interaction?

If YES: Extract it, test the pure part at Level 1, test integration at Level 2. DONE.

The pattern: factor tangled code into a pure function (validation, calculation, transformation) and a thin wrapper that calls the dependency. Test the pure function exhaustively at Level 1. Test the wrapper at Level 2 with real dependencies.

The fake repository was never needed.

3C: Glue/Orchestration Code - Can't Extract

The behavior IS the interaction with the dependency. The code exists to orchestrate, coordinate, or handle dependency behavior. You can't extract it without losing the thing you're testing.

Examples: retry logic, circuit breakers, saga orchestration, caching invalidation logic, rate limiting, multi-step workflows.

If you're here: Go to Stage 4.

If YES to all: Use real system at Level 2. DONE.

If NO to any: Go to Stage 5.

1. Evidence lives at Level 1
2. Code can't be factored into pure computation
3. Real system can't produce the behavior reliably/safely/cheaply/observably

Now and only now may you consider test doubles. But you must match a specific exception.

Exception 1: Failure Modes

When: You need to test behavior under specific failure conditions that the real system can't reliably produce.

Examples: timeouts at specific points, connection resets mid-stream, partial writes/reads, throttling/rate limits, retryable vs non-retryable error codes, DNS failures, TLS handshake failures.

Double type: Stub that returns predetermined errors.

Exception 2: Interaction Protocols

When: Correctness depends on the conversation pattern, not just input/output.

Examples: multi-step workflows (create → poll → finalize), pagination loops and cursor handling, transactional outbox patterns, sagas and compensating actions, "must call A before B", "must not call B if A failed", ensuring no extra calls (cost, rate limits).

Double type: Spy that records calls, or Mock for strict sequence assertions.

Exception 3: Time and Concurrency

When: You need deterministic control over timing and scheduling.

Examples: scheduled retries with jitter, token refresh races, lease renewal loops, debounce/throttle logic, "eventually consistent" wait loops, visibility timeouts.

Double type: Fake clock, controllable scheduler.

Exception 4: Safety

When: The real system is destructive or irreversible.

Examples: payment providers (charges/refunds), email/SMS sending, destructive admin APIs (delete, rotate keys), systems with strict quotas/billing.

Double type: Stub that records but doesn't execute.

Exception 5: Combinatorial Cost

When: You need 100+ scenarios and real system is too slow/expensive.

Examples: HTTP client behavior matrix (status codes × retry rules × idempotency × timeout budgets), queue consumer (redelivery × poison messages × DLQ routing × batch sizes), caching (TTL × stale-while-revalidate × negative caching × stampede protection).

Double type: Fake that can be configured for each scenario.

Note: Speed alone is not an exception. A test that takes 2 seconds with a real DB is fine. You need BOTH 100+ scenarios AND hours of runtime to justify this exception.

Exception 6: Observability

When: You need to verify details the real system can't expose.

Examples: "Did we include this header?", "Did we batch these operations?", "Did we send the idempotency key?", "Did we avoid N+1 calls?", "Did we handle pagination correctly?"

Double type: Spy that records request details.

Exception 7: Contract Testing

When: Third-party API you don't control, need to verify your serialization/parsing.

Examples: verify request serialization matches API spec, verify you can parse all documented response variants, pin behavior when provider changes subtly.

Double type: Contract stub that enforces expected format.

If No Exception Applies

STOP. You should not use test doubles.

Options:

1. Re-examine Stage 3B: Can you really not extract pure logic?
2. Test at Level 2: Use real dependencies
3. Accept no Level 1 test: Some glue code doesn't need unit tests

Not every function needs all four phases. Use judgment:

• Simple utilities: Phase 1 + 2 may be sufficient
• Complex algorithms: All four phases
• Glue code: Phase 1 only (integration tests handle the rest)

Phase 1: Typical Cases

Start with scenarios users encounter most often. What's the most common input? What does successful execution look like?

Phase 2: Edge and Boundary Cases

Test limits, special values, error conditions:

Phase 3: Systematic Coverage

For code with loops, state machines, or combinatorial behavior:

• Loop coverage: zero iterations, one iteration, multiple iterations, max iterations
• State transitions: all valid transitions, verify invalid ones are rejected
• Combinatorial testing: key combinations of independent parameters

Phase 4: Property-Based Tests

Define invariants that should hold for ALL valid inputs, then let the test framework generate inputs.

MANDATORY for these code types:

• Parsers (parse(format(x)) === x)
• Mathematical operations
• Serialization/deserialization
• Complex algorithms where edge cases are hard to enumerate

Common properties:

Property-based testing is NOT optional for the categories listed above. If you're testing a parser or serializer without property-based tests, you're not done.

Stub: Returns predetermined responses. Doesn't record calls or verify behavior—just provides controlled outputs. Use for testing how your code handles specific error responses, simulating failures, or replacing destructive operations.

Spy: Records calls for later verification. The primary purpose is to observe WHAT was called, not to control the response. Use for verifying request parameters, ensuring operations are batched, checking call ordering or count.

Fake: A simplified but working implementation. Unlike a stub (canned responses), a fake has real behavior—just simpler than production. Use for running many test scenarios at speed, or controlling time/scheduling.

Mock: Verifies that specific interactions happen in a specific order. The strictest form of test double. Use sparingly—only when exact sequence matters for correctness (saga compensation order, protocol requirements). Mocks couple tests to implementation details.

Dummy: A placeholder that satisfies a type requirement but is never actually used. If you need many dummies, the function may have too many dependencies.

Critical distinction:

• Mock (BAD default): Framework intercepts method calls on real objects
• Test double (OK when exception applies): You provide an alternative implementation via dependency injection

Anti-Pattern 1: Skipping the Evidence Question

Wrong thinking: "I see OrderService calls repository.save(), so I need to test OrderService with a fake repository."

Right thinking: "What evidence do I need? Evidence that orders persist correctly. A fake repository proves nothing about persistence. Test at Level 2 with real database."

Anti-Pattern 2: Calling Fakes "In-Memory Implementations"

Wrong: Creating a class with an array that records calls and returns canned data, then calling it an "in-memory implementation."

Right: An in-memory implementation is a REAL implementation that happens to use memory (like SQLite :memory:). It has real SQL parsing, real constraints, real transactions. An array that records calls is a fake.

Anti-Pattern 3: Faking for Speed

Wrong thinking: "Real DB is slow (2 seconds), so I'll fake it."

Right thinking: Speed alone is not an exception. You need 100+ scenarios AND hours of runtime (Exception 5). A 2-second test is fine. Don't optimize developer experience at the cost of test validity.

Anti-Pattern 4: Faking What Can Be Tested Real

Wrong: Creating a fake HTTP client instead of testing against a real HTTP server (local test server or docker container).

Right: If you're testing "does my code handle HTTP responses correctly?"—use a real HTTP server. The fake proves your code works with your imagination of HTTP, not actual HTTP.

Anti-Pattern 5: Testing Library Behavior

Wrong: Testing that argparse correctly parses the --verbose flag.

Right: Testing YOUR behavior that uses the parsed result (e.g., verbose mode produces detailed output with DEBUG prefix).

Trust the library. Test your code.

Failure 1: Skipped Stage 1 entirely

What happened: Agent saw code with dependencies and immediately created fakes for everything.

Why it failed: Never asked "what evidence do I need?" Created tests that pass but prove nothing about production behavior.

How to avoid: ALWAYS start at Stage 1. Write down the answer before proceeding.

Failure 2: Confused fake with real in-memory implementation

What happened: Agent created InMemoryUserRepository with array storage, claimed it satisfied "DI with real in-memory implementations."

Why it failed: An array is not a database. It doesn't enforce constraints, handle transactions, or validate SQL.

How to avoid: Real in-memory = actual database/system in memory mode. Fake = canned responses you wrote.

Failure 3: Used speed as sole justification

What happened: Agent said "Real DB is slow, so I'll fake it for faster tests."

Why it failed: Speed alone doesn't qualify for Exception 5. Need 100+ scenarios AND hours of runtime.

How to avoid: Ask: "Do I have 100+ scenarios? Would real system take hours?" If no to either, use real system.

Failure 4: Tested library instead of own code

What happened: Agent wrote tests verifying that Zod validates schemas correctly.

Why it failed: Tested the library, not the application. Zod works. Test what YOUR code does with validation results.

How to avoid: Ask: "Am I testing MY behavior or the library's behavior?" Trust libraries; test your code.

Failure 5: Used exception without documenting which one

What happened: Agent created test doubles but didn't note which exception case justified them.

Why it failed: No way to verify the doubles are legitimate. Future maintainers can't assess whether the exception still applies.

How to avoid: When using doubles, add a comment citing the specific exception (e.g., "Exception 1: testing timeout retry behavior").

Mocking (using mocking frameworks to intercept and stub arbitrary methods) gives you a test that passes while your production code fails. This is worse than no test at all.

If you feel you need to mock:

1. Redesign using dependency injection with real implementations or legitimate doubles (matching an exception case), OR
2. Test at a different level—push to Level 2 or 3 where real dependencies are available

The seven exception cases use test doubles (stubs, spies, fakes), not mocks. The distinction:

• Mock: Framework intercepts method calls on real objects
• Test double: You provide an alternative implementation via dependency injection

• Stage 1 question answered: "What evidence do I need?" is written down for each test
• Evidence exists at the level where it can be proven (not faked)
• No test doubles without matching exception case
• Exception case documented in comments if doubles are used
• Property-based tests present for parsers, serializers, math operations, and complex algorithms
• Tests verify behavior (assertions on outputs), not implementation (assertions on call counts)
• All tests pass