# Process Test Design

> Use when writing, reviewing, or pruning tests in ANY language - deciding unit vs integration vs e2e, whether to mock/patch or use the real dependency, which edge/adversarial inputs to cover (unusual UTF, emoji, CJK, binary, wrong types, oversized), why a test is flaky or order-dependent, or whether a test earns its keep. Keywords - mock, monkeypatch, spy, stub, fake, fixture, e2e, integration test, flaky, order-dependent, sleep, adversarial input, low-value test, coverage, pytest, vitest, jest, go test, cargo test, bats. For the red-green discipline see process-test-driven-development; for what to validate at a boundary see coding-input-sanitization.

- Skill: `bitranox/process-test-design` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add bitranox/process-test-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bitranox/process-test-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: bitranox (https://skillmd.com/u/bitranox)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/bitranox/process-test-design

---


# process-test-design

## Overview

How to write tests that are worth having: real behavior over mocks, the edge inputs that actually
break code, deterministic and order-independent, and no low-value filler. This is the WHAT-to-write
companion to `bitranox:process-test-driven-development` (the red-green-refactor discipline and the
mock anti-patterns) - follow that for WHEN/order; this for test design and quality.

**Core principle: a test must be able to fail for a real, specific reason.** A test that cannot fail
(asserts nothing, restates the implementation, or exercises a mock) is negative value - it adds
maintenance and false confidence. Test observable behavior at a boundary, not internals.

**Every principle here is language-neutral; only the mechanics differ.** Examples name Python where
one is needed, but the rules apply to any language - the per-ecosystem commands, seams, and tier
conventions are in "Per-language mechanics" below. When your language is not listed, map it by role
(what is the injection seam, what marks a test tier, what shuffles order) rather than assuming the
rule does not apply.

## Prefer real over mocked; mock only at the true edge

- **Default to integration / e2e against the real dependency** (real DB, real broker, real HTTP via a
  local server or recorded fixtures). In-memory fakes accept arguments the real service rejects, so
  green unit tests can still ship a broken contract. Treat the integration/e2e run as the proof.
- **Avoid monkeypatching. Use dependency injection.** Pass the collaborator in (a port/protocol) and
  substitute a real-ish fake or the real thing in tests. Reach for `monkeypatch`/patch only at a true
  external edge you cannot inject (a third-party global, the clock, the network) - never to reach into
  your own internals. Patching your own code is a design smell: make it injectable instead.
- **Fakes live behind the same interface as the real thing** (a `fake_*` implementation of the port),
  exercised by the same contract tests as the real adapter, so the fake cannot drift.
- **"Patch the network/clock" is the fallback, not the choice, when you own the caller.** If the code
  under test is yours, the collaborator IS injectable - inject it, even for a unit test, and keep the
  real call for the integration tier. Patching a global (`fetch`, the HTTP module, the clock) is
  correct only when nothing you own sits between the test and that global: third-party code you cannot
  change calls it, or the call is buried in a dependency. "It is an external edge" does not license
  patching a global you could have passed in.
- See `bitranox:process-test-driven-development` -> `testing-anti-patterns.md` (testing the mock,
  test-only methods in production, incomplete mocks, integration-as-afterthought).

## Per-language mechanics

Same rules, different handles. The "self-mock tell" column is what to grep for when auditing a suite:
it is the idiom that reaches into the code under test instead of injecting at a seam.

| Language        | Inject the seam as                | Self-mock tell (the anti-pattern)                                | Real dependency for the e2e tier        | Order / shuffle                                                                                                                                                                                   |
|-----------------|-----------------------------------|------------------------------------------------------------------|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Python          | constructor arg typed `Protocol`  | `monkeypatch.setattr(MyClass, "_method")`                        | testcontainers, a local server, `respx` | `pytest -p randomly`                                                                                                                                                                              |
| TypeScript / JS | constructor arg or interface      | `vi.spyOn(obj, "privateMethod")`, `jest.mock` of your own module | `node:http` server, MSW, testcontainers | `vitest --sequence.shuffle`                                                                                                                                                                       |
| Go              | interface parameter               | package-level function var swapped in a test                     | `httptest.Server`, real DB in Docker    | `go test -shuffle=on`                                                                                                                                                                             |
| Rust            | generic param or `Box<dyn Trait>` | `#[cfg(test)]` branch inside production code                     | `wiremock`, a real DB behind a feature  | `cargo test -- -Z unstable-options --shuffle` (nightly only; `cargo nextest run` does NOT shuffle - it runs lexicographically, one process per test, and `RUST_TEST_SHUFFLE` is silently ignored) |
| Bash            | a command name/path variable      | redefining the function under test in the test                   | a stub binary earlier on `PATH`         | `bats` (isolate per file)                                                                                                                                                                         |

**Mark the tiers so a bare test run stays offline.** Every ecosystem needs a way to keep the
network/DB tests out of the default run; pick the project's and use it consistently: pytest markers
(`-m "not integration"`), a filename or directory convention (`*.e2e.test.ts`, `test/e2e/`), a
separate config or workspace project, Go build tags, Rust `#[ignore]`, a `bats` tag. State it in the
project's own docs so the split is discoverable - inventing a private convention per test file is
how the integration tier quietly stops running.

**A marker only skips when something acts on it - registering it is not wiring it.** In pytest,
listing a name under `[tool.pytest.ini_options] markers` silences the unknown-marker warning and
nothing more; the skip has to come from a `pytest_runtest_setup` hook in `conftest.py` that reads
`item.iter_markers()` and calls `pytest.skip()`. A marker that guards a CONDITION (a platform, a
capability, an available service) is the dangerous case, because a registered-but-unwired one reads
exactly like a guard and runs everywhere. Before you trust one, grep `conftest.py` for the marker
name; if only the config mentions it, it is documentation. Then prove the skip fires by forcing the
condition rather than assuming.

Watch for two families that mean the same thing - one wired, one not. A repo that carries both
(`posix_only`/`windows_only` wired to a real skip, and `os_posix`/`os_windows` registered and inert)
hands you a coin flip. Fix that at the root by wiring the unwired family, not by renaming the
markers in the one test that failed: the rename fixes today's file and leaves the trap armed for the
next test someone writes.

**What decides the tier is the dependency, not the realism.** A test belongs in the marked tier when
it needs something the machine does not already provide - an external service, a shared DB, Docker,
credentials, the real network - or when it is slow enough to hurt the default run. A dependency you
start and stop inside the test process (an ephemeral loopback HTTP server, a temp-file DB) is real
enough to prove the contract while staying offline, deterministic and fast, so it belongs in the
DEFAULT run. "Uses the real implementation" and "must be marked and skipped" are different
questions; answer them separately.

## Injecting a callable as a seam: two traps a green suite cannot show

Injecting a stdlib callable so a test can substitute it is the right shape. Two things about it
fail silently.

**Annotate the attribute with YOUR contract, not the stdlib's.** A bare assignment makes pyright
strict infer the library function's exact signature, and an honest double is then rejected:

```python
self._sleep = time.sleep                              # infers (seconds: _SupportsFloatOrIndex, /) -> None
self._sleep: Callable[[float], None] = time.sleep     # infers YOUR contract
```

Under `typeCheckingMode: strict`, assigning a double declared `(_seconds: float) -> None` to the
first form is an error - the parameter is positional-only and `_SupportsFloatOrIndex` is not
assignable to `float`. The second form accepts it. The failure lands on the TEST, so it reads as a
bad double rather than as a missing annotation.

**After adding a wait or retry to a shared path, read the test file's WALL CLOCK, not its pass
count.** Every pre-existing test that goes through that path now pays the real delay. One 30-second
settle took a file from 0.07s to 90.05s while staying fully green. A slow suite never fails; it
decays until someone stops running it. Assert on elapsed time in CI, or inject the clock too.

## Adversarial inputs at the boundary (the test side of sanitization)

For any function/endpoint at an application or facing-API boundary, test the input battery, not just
the happy path. (Validation rules: `bitranox:coding-input-sanitization`; full per-codepath matrix:
enumerate every variant/caller a path serves and cover each branch.)

| Axis           | Cover at least                                                                                       |
|----------------|------------------------------------------------------------------------------------------------------|
| Text / Unicode | empty, whitespace-only, very long; non-ASCII, accented, combining marks, RTL, zero-width, emoji, CJK |
| Bytes          | control chars, NUL byte, invalid UTF-8 / raw binary                                                  |
| Type           | wrong type (str where int, None, list where scalar), missing field, extra field                      |
| Numbers        | 0, -1, 1, max, off-by-one at every limit, overflow, NaN/inf where float                              |
| Size           | empty collection, one element, at the cap, over the cap (DoS bound)                                  |
| Structure      | malformed JSON, truncated payload, duplicate keys, deeply nested                                     |

Assert the SPECIFIC behavior (rejected with a typed error, normalized, or escaped) - not just "does
not crash".

**How much is enough (the stopping rule).** Cover one case per BRANCH the code actually takes, not
one per input you can imagine. For each failure the code distinguishes - a different error type, a
different message, a different recovery - there is one test. Inputs the code handles identically get
one representative case between them. Applied to a thin HTTP wrapper: one non-2xx case if every
status is treated the same (two if 4xx and 5xx diverge), one malformed-body case, one
transport-failure case, and no more. When a caller only ever sees your declared error type, the test
that matters is the one asserting the raw underlying error was converted to it.

**The branch rule governs the axis table above.** An axis with no corresponding branch in the code
gets no test - if nothing bounds a number, an "at max / overflow" case asserts nothing you have
promised. But do not just drop it: a missing branch where the axis clearly applies (an unbounded
size or length reaching a real sink) is a CODE finding - report the absent validation rather than
writing a test that documents its absence.

## Scrub a captured artifact fully before it becomes a fixture

A captured artifact (a packet capture, a protocol exchange, a log or config dump) committed as a
test fixture carries more than its headers. A scrub that only touches the header/summary fields
looks complete while the structured payload underneath - options, TLVs, nested records - still
carries site topology (internal hostnames, domain names, subnets, device identifiers, vendor/serial
data). This is general practice, not tied to one protocol: it applies to DHCP, DNS, LLDP, and SNMP
captures, and equally to log and config dumps - scrub every layer the format defines, not just the
fields visible in a summary view.

**Assert the shipped fixture's fields in a test**, not only once by eye before committing. A fixture
nobody asserts on can silently re-acquire unscrubbed content the next time it is regenerated (a
re-capture, a re-export): the eyeball check does not repeat, the assertion does. Parse the committed
fixture and assert the sanitized values, including the structured payload fields, so the scrub is
enforced by the suite rather than remembered by a person.

## Deterministic and order-independent

- **No dependence on test execution order.** No shared mutable module/global state between tests; each
  test sets up and tears down its own world (fixtures). A test must pass run alone and in any order
  (green with random ordering on and off).
- **No real `sleep` for timing.** Poll a condition with a timeout (condition-based waiting), or inject
  the clock. A fixed `sleep(n)` is either flaky (too short) or slow (too long).
- **Inject time and randomness.** No bare now-clock, RNG, or UUID call in code under test - pass a
  clock / seed so the test is reproducible. (`datetime.now()`/`random`/`uuid4` in Python;
  `Date.now()`/`Math.random()`/`crypto.randomUUID()` in JS, or `vi.useFakeTimers()` at the edge;
  `time.Now`/`rand` in Go; `Instant::now`/`rand` in Rust; `$RANDOM`/`date` in Bash.)
- **No unmarked network / filesystem / external resource.** Those belong in integration tests
  (marked, opt-in), not the unit suite. The unit suite runs offline and identically every time.
- A flaky test is a bug in the test or the code, never "just re-run it" - fix the determinism.

## Run in a clean, project-correct environment

**The rule in any language: run against the project's OWN pinned toolchain, resolved from a lockfile,
never an ambient or global one.** Two runs that resolve different dependency versions turn a real
defect into "works on my machine" and an environment flake into a phantom bug. Per ecosystem: commit
the lockfile and install from it, not from the loose ranges (`uv sync` / `npm ci` not `npm install` /
`go mod download` with a committed `go.sum` / `cargo build --locked`), and pin the runtime version
(`requires-python`, `.nvmrc` or `engines`, the `go` directive, `rust-toolchain.toml`). An
environment-shaped failure - a missing module you know is installed, a flood of phantom type errors,
audit findings for packages not in your tree - is a wrong-environment smell to verify before you
trust it as a code bug.

Python specifics, where the trap is most common:

- **Run tests in the project's OWN venv, never the IDE's.** An ambient `VIRTUAL_ENV` (PyCharm, or carried
  over from another project's shell) silently hijacks the interpreter, so the suite runs against the
  wrong env. Isolate it: `env -u VIRTUAL_ENV uv run pytest` (mechanism + the bmk variant:
  `bitranox:coding-python-uv` "stray VIRTUAL_ENV"). "Fresh" = the project venv, isolated - only recreate
  (`uv venv --clear && uv sync`) when debugging suspected env corruption.
- **A wrong-venv failure masquerades as a code failure.** `ModuleNotFoundError` for a dep you know is
  installed, a flood of phantom type-check errors, or pip-audit CVEs for packages not in your tree are a
  WRONG-VENV smell, not a real defect. Before trusting such a failure, verify the interpreter:
  `uv run python -c "import sys; print(sys.executable)"` should point at `./.venv`. (Evidence before
  conclusions - see `bitranox:process-review-verification-before-completion`.)
- **Keep `.venv` out of git.** The project venv is a per-machine build artifact - never commit it;
  gitignore it (and untrack it if it slipped in). Mechanics: `bitranox:compuse-git` "Don't track local
  build artifacts".

## Prune low-value tests

Delete a test when it:
- asserts nothing (or only that no exception was raised, for logic that should assert a result),
- restates the implementation line-by-line (changes whenever the code changes, catches nothing),
- tests the language, framework, or a mock rather than your behavior,
- duplicates another test's coverage with no new branch.

Fewer, behavior-focused tests beat many brittle ones. Coverage percent is a smell detector, not a goal.

**Never commit a test that asserts behavior the code does not have.** When docs promise something the
code never wired up (a documented cache nothing calls, a flag with no effect), the honest output is a
FINDING - the feature is missing or the doc is wrong - not a red test left in the suite. A permanently
failing test is a broken gate, and a skipped one is a comment that rots. Report the gap, and write the
test when the behavior lands.

## A gate that only ever passed has not been shown to gate

A regression/perf gate that compares each run ONLY against a committed baseline answers "did this
move", never "is this right". A cell already broken when the baseline was seeded passes forever, and
a slow bleed of sub-threshold regressions never trips it. Pair the relative check with an ABSOLUTE
sanity bound (or a sibling/reference cell measured in the SAME run), declare a known-bad cell with a
tracking id so it is reported rather than frozen in, and make a declared cell that RECOVERS fail too
so a stale waiver cannot re-hide a fix.

The same doubt applies to any gate: validate it against a KNOWN-BAD input once, so you have seen it
go red. Otherwise "green for months" is equally consistent with "it stopped gating months ago".

**An exemption or scope limit added to a guard, filter, or validator needs the same proof, run in
the untouched direction.** Add a test where the trigger is present but the exemption must NOT
apply, asserting the verdict stays unchanged - not only a test proving the exemption fires where it
should. Then keep the case the guard was built to catch as a permanent test, re-run after every
exemption added later: two narrow-looking exemptions added in the same change can each pass alone
and still add up to silencing the guard on its own motivating case.

Four mechanics worth stating because they silently defeat a test that looks right:

- **`from m import X` binds a COPY at import time.** `monkeypatch.setattr(m, "X", v)` mutates `m`'s
  namespace and never reaches the consumer's already-bound name, so the test runs against the
  original value and passes for the wrong reason. Patch at the CONSUMER (`setattr(consumer, "X")`)
  or, better, inject - which is why this skill prefers a seam over patching in the first place.
- **A correctness test cannot prove an INDEX is used.** Results rank correctly whether the store
  used the index or full-scanned. Assert the plan (`EXPLAIN` shows an index scan, `list_indices` is
  non-empty), or the check silently passes on an O(n) path.
- **Two files sharing a BASENAME collide, and the loser's tests exercise the winner.** Python
  resolves an import to the first match on `sys.path`, and pytest's default `prepend` mode prepends
  each test root as it collects, so with two same-named modules the one collected FIRST wins for the
  whole run. The other directory's tests then run against a file you never changed. Both suites stay
  green, so nothing announces it; the tell is a fix that reads as absent in the full run while
  passing in isolation, and the shadowed file's real coverage is zero. Ship the module once, or run
  `--import-mode=importlib`, which keys modules by path instead of basename.
- **A teardown on a doctest's LAST line does not run when an earlier line raises.** A
  `mock.patch(...).start()` closed only by a trailing `mock.patch.stopall()` leaks the patch into
  every test collected after it, and under `--doctest-modules` the doctest executes in the collected
  module's namespace, so the leak is global. The resulting failures land far away and look unrelated
  to the doctest. Prefer a real injectable seam over patching inside a doctest at all.

Prove a codec/serializer swap by SEMANTIC equivalence - decode the new output with the OLD decoder
and compare the values - not by byte-equality, which fails on harmless ordering or padding changes
and passes on a compatible-looking but wrong encoding.

**A check built on a lenient parser can only see what that parser exposes.** A validator that reads
structured input through a forgiving reader - a bare `text.split("---", 2)[1]`, an unanchored regex,
a try/except that swallows a parse error - inherits every malformation that reader tolerates. A
second, smuggled copy of the block further down the file lands wherever the split throws its
unexamined remainder, so a check that only asserts on the fields it extracted reports the file clean
while never looking at what the parser silently dropped. Before trusting such a check, enumerate
concretely what its own parser forgives (a duplicated block, a glued-on closing delimiter, a
duplicate key a dict-based loader overwrites), then verify each one STRUCTURALLY against the raw
text - a delimiter count, a line scan - rather than through the same parser, which by construction
cannot see what it already swallowed.

## Quick checklist

- [ ] Real dependency or an injected fake behind the real interface; patch only where you own no seam
- [ ] Integration / e2e path exists and is the proof of the contract
- [ ] Test tiers marked so the default run stays offline, by the project's stated convention
- [ ] Boundary inputs covered (UTF/emoji/CJK/binary/wrong-type/oversized/edge numbers), asserting specific behavior
- [ ] Every error branch the code can raise has a test asserting the declared error type
- [ ] Order-independent (passes alone and shuffled); no shared mutable state
- [ ] No real `sleep`; time/randomness injected; unit suite offline
- [ ] Runs against the project's own locked toolchain, not an ambient/global one; an environment-shaped failure is a wrong-env smell, not a code bug (Python: `env -u VIRTUAL_ENV uv run ...`)
- [ ] One behavior per test; name states the behavior
- [ ] No test that cannot fail for a real reason

## Common mistakes

- **Mocking your own internals** instead of injecting them. Make the seam a port; pass a fake in.
- **Patching a global you could have injected.** "The network is an external edge" is not a licence to
  stub `fetch` when the caller is your own code - inject the collaborator and keep the real call for e2e.
- **Inventing a private tier convention per file.** If the project has no stated way to mark
  integration tests, add one and document it; an undiscoverable tier is a tier that stops running.
- **Green units, broken contract.** Fakes accepted what the real service rejects. Add the integration test.
- **"It passes on my machine / re-run it."** Flakiness is a defect - fix order/timing/clock, do not retry.
- **Testing the happy path only.** The bugs live in the edge battery above.
- **Chasing 100% coverage** with assertion-free or impl-mirroring tests. Delete those; they hide rot.
- **Single-layer mutation stays green (defense in depth).** Disabling ONE validation check can leave its test green because a LATER check rejects the same bad input, so the test looks like coverage it lacks. Mutate the single layer first; if it stays green, find which later check absorbed it - the test is asserting a contract no single mutation can break, so prove it by disabling the whole defense stack in ONE mutation. Copy the file aside before mutating and restore from that copy, never `git checkout -- <file>` - it restores from the index or HEAD and so discards the uncommitted work you are testing.

## Cross-references

- `bitranox:process-test-driven-development` - red-green-refactor discipline + `testing-anti-patterns.md`.
- `bitranox:coding-input-sanitization` - what to validate/escape at the boundary (this skill is how to TEST it).
- Per-repo convention (bmk): `make test` (unit, offline) vs `make testintegration` (real resources);
  shared fixtures in `tests/conftest.py`.