Loom Test Strategy

Test Strategy

Overview

How much to test, at what altitude, and in what order — balancing signal against cost. This file owns the pyramid, coverage goals, risk prioritization, and flaky-test diagnosis. For writing tests and test-double choice see loom-testing; for browser E2E see loom-e2e-testing.

The Pyramid and the Altitude Tradeoff

Push tests down to the cheapest altitude that still exercises the risk. Each layer up costs more to write, run, and debug, and fails for more unrelated reasons.

⚠ Ice-cream cone anti-pattern: inverted pyramid — many slow E2E tests, few unit tests. Symptoms: hours-long CI, chronic flakiness, "just re-run it" culture, hours to localize a failure. Cause: E2E is easy to start (record-and-play) but the suite becomes unmaintainable. Fix: for every E2E failure, ask "what unit/integration test should have caught this?" and push it down.

⚠ Testing Trophy (Kent C. Dodds) is a legitimate variant for UI/front-end-heavy apps: fewer isolated units, a fat integration middle (component + real collaborators via Testing Library), thin E2E. Choose by where your risk and refactor-churn live — do not cargo-cult 70/20/10.

Coverage — a floor, never a target

Coverage measures what is untested; it says nothing about assertion quality. Mandating 100% breeds assertion-free tests that run lines to hit the number.

• Track branch, not just line — line coverage passes while else/error arms stay untested.
• Mutation testing (mutmut/cosmic-ray, cargo-mutants, StrykerJS) is the real quality gate: it mutates code and asserts a test fails. Surviving mutants expose weak assertions that 100% line coverage hid. Run periodically on core logic, not every commit (it's slow).
• Set a coverage ratchet (fail if coverage drops) rather than a fixed high bar — prevents rot without incentivizing junk tests.

What to Test vs Skip

Always: business/domain rules, input validation and error handling, security-sensitive ops, data transformations, state transitions, boundary/edge cases, and a regression test for every fixed bug.

Usually skip: trivial getters/setters, framework-generated or third-party code, config constants, pure pass-throughs, logging (unless the log is a contract).

// BAD — asserts nothing meaningful
test("getName returns name", () => expect(new User("Jo").getName()).toBe("Jo"));
// GOOD — asserts a rule that can actually regress
test("name cannot be blank", () =>
  expect(() => new User("Jo").setName("")).toThrow(ValidationError));

Risk-Based Prioritization

Test effort follows impact × likelihood, not uniform coverage.

Risk drivers: revenue/compliance/data-integrity impact, code complexity, change frequency (hot files), historical bug density, dependence on flaky externals.

Map priority to cadence so fast feedback stays fast:

• P0 (every commit): auth/authz, payments, data integrity.
• P1 (PR merge): core workflows, API contract tests.
• P2 (nightly): edge cases, performance.
• P3 (weekly): back-compat, deprecated paths.

Categorization & CI Wiring

Tag suites so CI can select by layer/priority:

describe("[unit][fast] UserService", () => {});
describe("[integration][slow] UserRepo", () => {});
describe("[e2e][critical] Checkout", () => {});
// npm test -- --grep="\[unit\]"    (or vitest --project unit / jest projects)

Fastest-first, fail-fast pipeline:

jobs:
  unit:                       # every push — seconds
    steps: [{ run: "npm test -- --grep='\\[unit\\]' --coverage" }]
  integration:               # needs services
    needs: unit
    services: { postgres: { image: postgres:15 } }
    steps: [{ run: "npm test -- --grep='\\[integration\\]'" }]
  e2e:                       # slowest, last
    needs: integration
    steps: [{ run: "npm run test:e2e" }]

CI hygiene: cache deps/build, shard large suites across machines (--shard=1/4), run only affected tests locally (--changedSince=origin/main), and treat a coverage drop as a failure.

Domain Strategies (layered priorities)

Concrete assertion examples for each domain live in loom-testing; here is what to cover at which priority.

API: contract tests (schema, status codes, error shapes, authz) P0 → business logic P0 → integration (DB, external svc, rollback) P1 → perf/rate-limit P2. Contract tests are the cheapest guard against silent breaking changes — pin request/response schemas.

Data pipeline: data-quality (schema, types, nulls, dupes) P0 → transformation correctness + no data loss P0 → source/sink integration + idempotency P1 → throughput/memory on large sets P2.

ML model: data validation + leakage detection + split integrity P0 → behavioral (invariance, directional, min-functionality) P0 → quality thresholds + fairness across groups + regression vs baseline P1 → serving/contract/versioning P1. Assert on behavior and threshold floors, not a single accuracy number.

Infrastructure: IaC validation (syntax, security policy, naming, cost) P0 → deploy smoke + health checks P1 → observability (metrics/logs/alerts fire) P1 → resilience/chaos (restart, partition, exhaustion) P2. Assert against the plan (e.g. no 0.0.0.0/0 ingress) before spending on live deploys.

Flaky-Test Diagnosis (canonical)

A flaky test destroys trust in the whole suite. Quarantine it (don't delete), then fix the root cause — never paper over with a blind retry.

Diagnose:

# reproduce: loop until it fails, then bisect the cause
for i in $(seq 1 200); do npm test -- TestName || { echo "fail at $i"; break; }; done
npm test -- --shuffle           # order dependence?
npm test -- --maxWorkers=4      # parallel race?

Always-fails-at-same-point ⇒ it's a real bug, not flaky. Then instrument (verbose logs, timing, failure artifacts), fix the root cause, and verify by running 1000× locally + several CI runs before un-quarantining.

Prevention checklist:

• Deterministic data (seeded, no bare random()/Date.now())
• Async uses explicit condition waits, never sleep/waitForTimeout
• Unique resource names per test (UUID)
• Teardown always runs (try/finally, afterEach)
• No hardcoded timing (sleep(100) is a smell)
• External services mocked/faked
• Clocks injected/frozen
• Order-independent (passes shuffled)
• Shared state reset between tests
• Environment reproducible (containerized, pinned)

Test Infrastructure

Ephemeral, isolated environments. Spin dependencies per-run; use tmpfs/in-memory volumes for speed.

# docker-compose.test.yml — DB on tmpfs, throwaway
services:
  test-db:
    image: postgres:15
    environment: { POSTGRES_DB: test, POSTGRES_PASSWORD: test }
    tmpfs: ["/var/lib/postgresql/data"]   # RAM-backed → fast, no cleanup

Test data via factories, not shared fixtures — each test mints unique data (sequence/UUID), avoiding cross-test coupling:

class UserFactory {
  private seq = 0;
  create(o: Partial<User> = {}): User {
    const n = ++this.seq;
    return { id: `user-${n}`, email: `user${n}@test.com`, role: "user", ...o };
  }
}

Parallelization — pick isolation to match:

⚠ In CI cap workers (--maxWorkers=50%); over-subscribing CPUs makes timing-sensitive tests flake. Longer testTimeout in CI than local.

Best Practices

• Behavior over implementation — a passing suite should survive any refactor that preserves behavior.
• Independent — each test owns its setup; any order; no shared mutable state.
• Right double for the job — stub inputs, mock only contractual interactions, fake awkward collaborators (taxonomy in loom-testing).
• Fast feedback — optimize the local unit loop; slow tests move to nightly, not the commit path.
• Same quality bar as prod code — refactor tests, delete obsolete ones, keep intent in the name.

Verify before done

• Shape justified (pyramid vs trophy) by where risk/refactor-churn lives — not cargo-culted
• Every layer covers something the layer below can't — no redundant E2E for logic testable as a unit
• Coverage is a ratchet/floor with branch tracked; core logic spot-checked with mutation testing
• P0 set (auth, payments, data integrity) runs on every commit and is fast
• Each regression-worthy bug has a dedicated test
• Flaky tests quarantined + root-caused, not blind-retried; prevention checklist applied
• CI runs fastest-first, fail-fast, with coverage-drop gate