Design Testing Strategy

Design Testing Strategy

A reference manual for designing a fit-for-purpose, fit-for-criticality testing strategy.

This skill is decision-oriented, not philosophical: every gate is deterministic (ON when X / OFF when Y), every schema is enforced (field ordering matters), every example is worked end-to-end.

How To Use This Skill

1. Read Decision Gates in order (Gate 0 -> Gate 6). Each gate is independent — you may finish with any subset of test types ON.
2. Apply Strategic Skip Heuristics to remove ON gates that would yield low ROI for this artifact.
3. For each ON gate, fill the Test Matrix Schema (selected_types entry) — the field order is load-bearing.
4. List rejected types in rejected_types and deliberate skips in deliberately_skipped.
5. Produce a Test Cases to Cover markdown bullet list using ISTQB techniques from Case Design Techniques.
6. Cross-check against the matching Worked Example (A pure function / B HTTP+DB endpoint / C UI component).

Decision Gates

Apply gates in numeric order. Each gate produces an independent boolean (applies: true|false). Gates do NOT veto each other — a single artifact may have unit + integration + contract + property-based all ON.

Gate Application Algorithm

for gate in [Gate 0, Gate 1, ..., Gate 6]:
    if gate.ON_condition_met(artifact):
        result[gate.type] = applies: true
    else:
        result[gate.type] = applies: false

if Gate 0 is true:
    short-circuit: emit empty selected_types, document criticality=NONE, stop

Criticality Scale (used by Gates 3 and 6):

Test Type Reference

Google Test Size Mapping

Google Test Sizes (Bland) and SWE at Google Ch.11 classify tests by resources (size), independent of scope (paths covered):

A test's type (unit/integration/e2e) and size (small/medium/large) are orthogonal: a small integration test (Testcontainers Postgres in same process via JDBC) is legitimate.

Playwright vs Cypress (UI e2e)

Case Design Techniques

Use ISTQB Foundation Level black-box techniques to derive what to test inside each chosen test type. References: ISTQB BVA white paper, ASTQB black-box techniques.

1. Equivalence Partitioning (EP)

Divide input domain into partitions where the system is expected to behave the same way; ONE test per partition is sufficient.

Worked example — discount(orderTotal: number) -> number:

Four tests cover all partitions. EP alone misses boundaries — combine with BVA.

2. Boundary Value Analysis (BVA)

Bugs cluster at boundaries. For every boundary value B, test B-1, B, B+1 (or for floats, the smallest representable step).

Worked example — same discount function, boundary at 100:

Repeat for boundary at 500: test 499, 500, 501. Total: 6 boundary tests + 4 EP tests = 10 cases.

The B-1 / B / B+1 triplet has the same shape across boundaries (vary input, vary expected output, identical assertion); this is a natural fit for a table-driven test (see sub-section 5 below).

3. Decision Tables

When output depends on combinations of conditions. Each column is a rule.

Worked example — canCheckout(cartHasItems, paymentValid, addressOnFile):

Four tests, one per rule (* = don't care, dropped via merging).

4. State Transition

When behavior depends on history. Identify states, events, and forbidden transitions.

Worked example — Order state machine with states {draft, submitted, paid, shipped, cancelled}:

Cover one test per legal transition + one per forbidden transition (negative path).

5. Table-Driven Tests

When EP, BVA, or decision-table analysis yields 3+ cases with the same shape (same setup, same assertion, only inputs and expected outputs differ — e.g., parsing valid/invalid date formats; computing tax across brackets; routing rules) collapse them into a single table-driven test. The cases become rows in a data table; the test body iterates the rows and runs one assertion per row. References: Dave Cheney, Prefer table-driven tests; Go wiki: TableDrivenTests.

Do NOT force a table when setup, framework calls, or the assertion shape varies substantially across cases. Forced uniformity hides real differences behind a single name and produces obscure failure messages — keep those as separate, individually named tests.

Worked example — six EP+BVA cases for discount(orderTotal) (boundary at 100) collapsed into one table-driven unit test (TS / vitest syntax; the same pattern applies to Go t.Run, JUnit @ParameterizedTest, pytest parametrize):

describe("discount", () => {
  const cases: Array<{ name: string; input: number; expected: number }> = [
    { name: "EP: below threshold (typical)", input: 50,  expected: 0    },
    { name: "BVA: B-1 at boundary 100",      input: 99,  expected: 0    },
    { name: "BVA: B at boundary 100",        input: 100, expected: 0.05 },
    { name: "BVA: B+1 at boundary 100",      input: 101, expected: 0.05 },
    { name: "EP: mid tier (typical)",        input: 250, expected: 0.05 },
    { name: "EP: top tier (typical)",        input: 1000, expected: 0.10 },
  ];

  for (const c of cases) {
    it(c.name, () => {
      expect(discount(c.input)).toBe(c.expected);
    });
  }
});

The name column is mandatory: each row must produce an individually addressable test so failures point to the specific case, not "row 3 of 6". Rows that need a different assertion (e.g., the negative-input case throws) stay as separate tests outside the table.

Dependency Decision

For Gate 2 (Integration) and Gate 3 (Component/E2E), choose dependencies deliberately. The goal is maximum realism that still runs deterministically in CI.

Tradeoff summary: Testcontainers > in-memory fake > mock, but cost goes the same direction. Pick the cheapest level that doesn't lie about the boundary's behavior.

Strategic Skip Heuristics

Explicit "don't bother" rules. Skipping these is not laziness — it is risk-adjusted ROI per ISO/IEC/IEEE 29119 risk-based testing and Risk-Based Testing.

Test Matrix Schema

Every test strategy MUST be expressed as the YAML block below. Field ordering inside each list entry is load-bearing — judges and downstream tools parse the first key as the critical one (rationale / reason / why), and the second key as the categorical one (type / what).

Schema

test_strategy:
  artifact: "<path or short identifier>"
  rationale: "Why this test strategy is being applied to this artifact (specific, evidence-based)"
  criticality: "NONE | LOW | MEDIUM | MEDIUM-HIGH | HIGH"

  selected_types:
    - rationale: "Why this type is being applied to this artifact (specific, evidence-based)"
      type: "unit | integration | component | e2e | smoke | contract | property-based"
      size: "small | medium | large | enormous"
      framework: "vitest | jest | pytest | go test | JUnit | playwright | cypress | pact | hypothesis | ..."
      dependencies:
        - "List of dependencies: real Postgres via Testcontainers, in-memory fake, mocked HTTP via nock, etc."
      gate: "Gate N (the gate that triggered this selection)"

  rejected_types:
    - reason: "Why this type does NOT apply to this artifact (cite Strategic Skip Heuristic or gate that did not trigger)"
      type: "unit | integration | component | e2e | smoke | contract | property-based"

  deliberately_skipped:
    - why: "Cost / risk justification for skipping despite a partial signal"
      what: "A specific category of test cases being skipped (e.g., 'browser compatibility on IE11', 'load testing beyond 100 RPS')"

Worked YAML Example

test_strategy:
  artifact: "POST /users (user registration endpoint)"
  rationale: "User registration is a critical user-facing path; can be used by web and mobile apps independently of each other."
  criticality: "MEDIUM-HIGH"

  selected_types:
    - rationale: "Endpoint contains validation logic (email format, password rules, uniqueness) — Gate 1 ON for branch coverage"
      type: "unit"
      size: "small"
      framework: "vitest"
      dependencies: ["in-memory user repository fake"]
      gate: "Gate 1"
    - rationale: "Endpoint writes to Postgres and emits user.created event to Kafka — Gate 2 ON, real boundary behavior matters"
      type: "integration"
      size: "medium"
      framework: "vitest + supertest + Testcontainers"
      dependencies: ["Postgres 15 via Testcontainers", "Kafka via Testcontainers"]
      gate: "Gate 2"
    - rationale: "Consumed by mobile app and web app on independent deploy cadences — Gate 4 ON, prevents drift"
      type: "contract"
      size: "medium"
      framework: "Pact"
      dependencies: ["Pact broker"]
      gate: "Gate 4"

  rejected_types:
    - reason: "No UI surface in this artifact — Gate 3 OFF"
      type: "component"
    - reason: "No UI surface — Gate 3 OFF; e2e covered by web/mobile apps separately"
      type: "e2e"
    - reason: "Input domain (email, password) is large but invariants are well-covered by EP+BVA at unit level — property-based ROI is low at MEDIUM-HIGH criticality, only triggers Gate 6 partially"
      type: "property-based"

  deliberately_skipped:
    - why: "Project does not have post-deploy probe pipeline yet; smoke would be no-op"
      what: "Smoke test for /users after deploy"
    - why: "Non-functional load testing is out of scope for this task; tracked separately in performance backlog"
      what: "Load test verifying p99 < 200ms at 1000 RPS"

Field ordering checklist (judges check this verbatim):

• test_strategy: artifact BEFORE rationale BEFORE criticality.
• selected_types[*]: rationale BEFORE type BEFORE size BEFORE framework BEFORE dependencies BEFORE gate.
• rejected_types[*]: reason BEFORE type.
• deliberately_skipped[*]: why BEFORE what.

Case Listing Schema

After the matrix, produce a flat markdown bullet list of test cases to be implemented. This is separate from the YAML matrix because:

• a. it lists what to test, not how
• b. it links back to acceptance criteria

Format

## Test Cases to Cover

### AC-N: [criterion title]
- [type] description 
- [type] description 

### AC-N: [criterion title]
- [type] description 
- [type] description 

Where:

• type matches one of selected_types[*].type from the matrix
• description follows AAA / Given-When-Then (Dan North BDD) shape — see Bill Wake AAA (2001)
• AC-N references the acceptance criterion the case verifies (omit if non-AC-bound, e.g., infrastructure smoke)

Worked Example

## Test Cases to Cover

### AC-1: Discount returns the correct percentage based on the total
- [unit] discount returns 0% when total = 0 [EP partition: below threshold]
- [unit] discount returns 0% when total = 99 [BVA: B-1 at boundary 100]
- [unit] discount returns 5% when total = 100 [BVA: B at boundary 100]
- [unit] discount returns 5% when total = 101 [BVA: B+1 at boundary 100]

### AC-2: Discount fails when total is invalid
- [unit] discount throws when total = -1 [EP partition: invalid]

### AC-3: /orders saves the order to the database
- [integration] POST /orders persists order to Postgres and returns 201 with order id

### AC-4: /orders rejects duplicate idempotency key
- [integration] POST /orders rejects duplicate idempotency key with 409

### AC-5: /orders/:id returns order by id
- [contract] GET /orders/:id returns schema matching mobile-app pact

Sources & Further Reading

These 14 sources back every gate and rule above. When in doubt, consult the source linked at that gate.

1. Test Pyramid — Mike Cohn (2009, Succeeding with Agile) + Ham Vocke, The Practical Test Pyramid, martinfowler.com.
2. Testing Trophy — Kent C. Dodds (2018), The Testing Trophy and Testing Classifications and Write Tests.
3. Google Test Sizes — Mike Bland (2011), Small / Medium / Large; Software Engineering at Google Ch.11; Test Sizes (Google Testing Blog).
4. Google Testing on the Toilet — What Makes a Good End-to-End Test, Testing UI Logic - Follow the User, Origins (Mike Bland).
5. ISTQB Foundation Level — Black-box techniques: Boundary Value Analysis white paper; ASTQB Black-Box Techniques.
6. ISO/IEC/IEEE 29119 — Risk-based test process standard. Wikipedia overview.
7. Kent Beck — Test Driven Development: By Example (Addison-Wesley, 2002). Publisher page. ISBN 978-0321146533.
8. The Pragmatic Programmer (20th Anniversary Edition) — Hunt & Thomas (2019). pragprog.com.
9. AAA pattern — Bill Wake (2001), 3A — Arrange, Act, Assert. Given-When-Then — Dan North, Introducing BDD.
10. Property-based testing — Hypothesis: What is property-based testing?; QuickCheck (Haskell), fast-check (TS).
11. Contract testing / Consumer-Driven Contracts — Pact docs; Pactflow CDC explainer.
12. Testcontainers — testcontainers.com.
13. Table-driven tests — Dave Cheney, Prefer table-driven tests; Go wiki: TableDrivenTests.
14. Risk-based testing — Risk Management During Test Planning (softwaretestinghelp.com).

Worked Examples

Each example shows:

• a. the artifact and acceptance criteria
• b. gate-by-gate walkthrough
• c. test_strategy YAML following the schema
• d. Test Cases to Cover list
• e. commentary on rejected types

Example A — Pure Helper Function: formatCurrency(amount: number, code: string): string

Artifact

function formatCurrency(amount: number, code: string): string;
// e.g. formatCurrency(1234.5, "USD") -> "$1,234.50"
//      formatCurrency(1234.5, "EUR") -> "€1.234,50"

Acceptance criteria:

• AC-1: USD output uses $ prefix, comma thousands, period decimal, two decimal places.
• AC-2: EUR output uses € prefix, period thousands, comma decimal, two decimal places.
• AC-3: Throws Error("Unknown currency code") for unsupported codes.
• AC-4: amount = 0 formats as "$0.00" / "€0,00".

Criticality: LOW (helper used in display only, no money movement here).

Gate Walkthrough

test_strategy YAML

test_strategy:
  artifact: "src/util/formatCurrency.ts"
  rationale: "Pure helper function used in display only; no money movement here."
  criticality: "LOW"

  selected_types:
    - rationale: "Pure logic with currency-specific branches and number formatting; EP+BVA on amount, decision table on currency code"
      type: "unit"
      size: "small"
      framework: "vitest"
      dependencies: []
      gate: "Gate 1"
    - rationale: "Amount domain is unbounded floats; invariant 'parseCurrency(formatCurrency(x, c)) ~= x' is stable; sparingly applied (1-2 properties) at LOW criticality"
      type: "property-based"
      size: "small"
      framework: "fast-check"
      dependencies: []
      gate: "Gate 6"

  rejected_types:
    - reason: "No I/O, no boundary, no collaborators - Gate 2 OFF"
      type: "integration"
    - reason: "No UI surface - Gate 3 OFF"
      type: "component"
    - reason: "No UI surface - Gate 3 OFF"
      type: "e2e"
    - reason: "Internal helper, not consumed across deploys - Gate 4 OFF"
      type: "contract"
    - reason: "Library helper, no deploy pipeline target - Gate 5 OFF"
      type: "smoke"

  deliberately_skipped:
    - why: "Locale list is finite (USD, EUR); exhaustive enumeration via decision table is sufficient and more maintainable than i18n property tests"
      what: "Property-based fuzzing of currency code beyond known list"

Test Cases to Cover

### AC-1: USD output uses `$` prefix, comma thousands, period decimal, two decimal places.
- [unit] formatCurrency(1234.5, "USD") returns "$1,234.50" [EP: typical USD]
- [unit] formatCurrency(0.01, "USD") returns "$0.01" [BVA: B+1 smallest non-zero]
- [unit] formatCurrency(-0.01, "USD") returns "-$0.01" [BVA: B-1 negative side]

### AC-2: EUR output uses `€` prefix, period thousands, comma decimal, two decimal places.
- [unit] formatCurrency(1234.5, "EUR") returns "€1.234,50" [EP: typical EUR]
- [property-based] for any non-NaN finite x in [-1e9, 1e9] and code in {USD, EUR}: parseCurrency(formatCurrency(x, code)) is within 0.005 of x [round-trip invariant]

### AC-3: Throws `Error("Unknown currency code")` for unsupported codes.
- [unit] formatCurrency(1, "XYZ") throws Error("Unknown currency code") [Decision table: unknown code]

### AC-4: `amount = 0` formats as `"$0.00"` / `"€0,00"`.
- [unit] formatCurrency(0, "USD") returns "$0.00" [BVA: B at amount=0]
- [unit] formatCurrency(0, "EUR") returns "€0,00" [BVA: B at amount=0 for EUR]

Why types were rejected: Helper has no boundaries (no integration), no UI (no component/e2e), is internal and library-style (no contract/smoke), and at LOW criticality the cost of additional test types far exceeds the benefit.

Example B — HTTP POST Endpoint with DB and Multi-Consumer: POST /users

Artifact

A user-registration endpoint that:

1. Validates request body (email format, password complexity, age >= 13).
2. Checks email uniqueness against Postgres.
3. Inserts user record (transactional).
4. Emits user.created event to Kafka.
5. Returns 201 with {id, email, createdAt}.
6. Returns 400 for invalid input, 409 for duplicate email.

Consumed by: mobile app (iOS/Android) and web app on independent deploy cadences.

Acceptance criteria:

• AC-1: Valid request returns 201 and persists user.
• AC-2: Invalid email format returns 400 with field-level error.
• AC-3: Password not meeting policy returns 400.
• AC-4: Duplicate email returns 409.
• AC-5: Successful registration emits exactly one user.created event.
• AC-6: Response schema is stable for mobile + web consumers.

Criticality: MEDIUM-HIGH (auth surface, identity domain, multi-consumer public API).

Gate Walkthrough

test_strategy YAML

test_strategy:
  artifact: "POST /users (user registration endpoint)"
  rationale: "User registration is a critical user-facing path; can be used by web and mobile apps independently of each other."
  criticality: "MEDIUM-HIGH"

  selected_types:
    - rationale: "Validators (email, password, age) are pure logic; EP+BVA on each field; one test per partition"
      type: "unit"
      size: "small"
      framework: "vitest"
      dependencies: ["in-memory user repository fake (for service-level unit if needed)"]
      gate: "Gate 1"
    - rationale: "Endpoint writes to Postgres and emits to Kafka; mocking these distorts transactional and ordering behavior - Testcontainers gives real boundary fidelity"
      type: "integration"
      size: "medium"
      framework: "vitest + supertest + Testcontainers"
      dependencies: ["Postgres 15 via Testcontainers", "Kafka via Testcontainers"]
      gate: "Gate 2"
    - rationale: "Public API consumed by mobile + web on independent deploy cadences; contract testing prevents schema drift breaking either consumer"
      type: "contract"
      size: "medium"
      framework: "Pact (provider verification)"
      dependencies: ["Pact broker", "consumer-published pacts from mobile and web"]
      gate: "Gate 4"
    - rationale: "Deployable HTTP service with a post-deploy pipeline; one minimal smoke verifies /users responds 201 in the deployed environment"
      type: "smoke"
      size: "large"
      framework: "Playwright (1 critical path)"
      dependencies: ["deployed environment URL", "test account seeding"]
      gate: "Gate 5"

  rejected_types:
    - reason: "No UI surface in this artifact - Gate 3 OFF; mobile and web repos own their own component tests"
      type: "component"
    - reason: "No UI surface - Gate 3 OFF; consumer e2e lives in mobile/web repos"
      type: "e2e"
    - reason: "Input domain is bounded and EP+BVA at unit level covers it; property-based on this glue endpoint adds infra without finding more bugs - Gate 6 OFF"
      type: "property-based"

  deliberately_skipped:
    - why: "Performance/load testing is out of scope here; tracked in dedicated performance backlog"
      what: "Load test verifying p99 < 200ms at 1000 RPS"
    - why: "Cross-region failover is owned by infrastructure team, not this endpoint"
      what: "Multi-region availability test"

Test Cases to Cover

### AC-1: Valid request returns `201` and persists user.
- [unit] validateEmail accepts "alice@example.com" [EP: well-formed]
- [integration] POST /users with valid body returns 201 and persists row in Postgres
- [smoke] POST /users in deployed environment returns 201 for a synthetic test account

### AC-2: Invalid email format returns `400` with field-level error.
- [unit] validateEmail rejects "alice@" [EP: missing domain]
- [unit] validateEmail rejects "" [BVA: empty boundary]
- [integration] POST /users with invalid email returns 400 and does NOT persist

### AC-3: Password not meeting policy returns `400`.
- [unit] validatePassword rejects 7-char password [BVA: B-1 at min length 8]
- [unit] validatePassword accepts 8-char password meeting policy [BVA: B at min length]
- [unit] validatePassword accepts 9-char password [BVA: B+1]
- [unit] validateAge rejects 12 [BVA: B-1 at boundary 13]
- [unit] validateAge accepts 13 [BVA: B at boundary 13]

### AC-4: Duplicate email returns `409`.
- [integration] POST /users with duplicate email returns 409 and does NOT emit event

### AC-5: Successful registration emits exactly one `user.created` event.
- [integration] POST /users emits exactly one user.created event to Kafka on success
- [integration] POST /users transaction rolls back when Kafka publish fails [State Transition: failure path]

### AC-6: Response schema is stable for mobile + web consumers.
- [contract] Provider satisfies mobile pact: POST /users response shape matches mobile contract
- [contract] Provider satisfies web pact: POST /users response shape matches web contract

Why types were rejected: No UI surface (component/e2e belong to consumer apps), bounded input space (property-based ROI low), out-of-scope concerns (load, multi-region) deliberately skipped with rationale.

Example C — UI Form Component: <RegistrationForm /> (web)

Artifact

A React form component:

1. Fields: email, password, confirmPassword, age.
2. Client-side validation: email format, password >= 8 chars with mixed case + digit, passwords match, age >= 13.
3. Submits to POST /users.
4. Shows inline field errors and submit-level errors (network, 409 duplicate).
5. Disables submit button while pending; re-enables on response.
6. WCAG 2.1 AA: labels bound to inputs, errors announced via aria-live, focus moves to first error on validation failure.

Acceptance criteria:

• AC-1: User can submit a valid form and is navigated to /welcome.
• AC-2: Invalid email shows inline "Enter a valid email".
• AC-3: Mismatched passwords show inline "Passwords must match".
• AC-4: Submit is disabled while request is in flight.
• AC-5: 409 response from server shows "This email is already registered" at form level.
• AC-6: Form is keyboard navigable; focus moves to first error on validation failure.
• AC-7: All inputs have programmatic labels; errors are announced via aria-live="polite".

Criticality: MEDIUM-HIGH (registration is a critical user-facing path; accessibility is regulated in many jurisdictions).

Gate Walkthrough

test_strategy YAML

test_strategy:
  artifact: "src/components/RegistrationForm.tsx"
  rationale: "React form component used in web app; registration is a business-critical user-facing path."
  criticality: "MEDIUM

…(truncated)