Test Fixture Realism

Test-Fixture Realism

Type: Open-source — client-agnostic methodology, no project-specific detail.

Created by akbarsha — https://github.com/iamakbarsha1

Distilled from a recurring cluster of test failures where the fixtures didn't match data reality or the assertion could pass without the code working.

Licence: Released under CC BY 4.0 — share and adapt for any purpose with credit. Full text: LICENSE at the repository root.

Feedback & Support: If a rule here proves wrong or needs sharpening, open an issue on the repository or contact the author at the profile link above. If the problem is the agent not following a rule below rather than the rule itself, that's an execution failure — acknowledge and correct it.

The core rule

A test proves something only if two things hold: its fixtures resemble the data the code meets in production, and its assertions cannot pass unless the code actually worked. Fabricated fixtures that skip referential integrity, aggregate assertions over a shared store, unseeded randomness, and mocks that echo the input all produce green tests that verify nothing. Build fixtures that are real enough to exercise the code, and assertions specific enough to fail when it breaks.

Checks

• Assert on the rows YOU created, not aggregates. Against a shared DB or store, other data and parallel tests pollute any count(*) or "all rows" assertion. Tag the rows your test inserts (a marker, a returned id) and assert on exactly those. (A test asserted a total row count; another test's seed data made it flap between pass and fail.)
• Fixtures need real referential integrity. A foreign key must point to a parent row that actually exists — create the parent, use its RETURNED id. A made-up id either errors or silently matches nothing, and the test then exercises an empty join. (A fixture used a hardcoded parent id; the join returned zero rows and the test asserted on emptiness without noticing.)
• Kill nondeterminism: seed randomness, pin the clock. Random fixture values and time-derived keys make tests flaky and can collide across runs. Seed the RNG and pass a fixed time in (see the inject-ambient-inputs skill), so the same inputs produce the same fixtures every run. (Fixtures keyed on the current time occasionally collided, producing an order-dependent failure.)
• A guard's fixture must fail closed. When testing a validation/guard, the fixture that VIOLATES it must actually trip the failure path — an omitted or malformed field should make the assertion fail, not pass by default. This is the fixture side of prove-the-test-can-fail. (A "reject when field missing" test used a fixture that still had the field, so it never exercised the rejection.)
• Watch mocks that load the real module or echo the input. Automock and partial mocks can pull in the real module and run its side effects; and a test whose expected value is computed from the same call it's testing (a search that echoes its own query) passes trivially. Assert against an INDEPENDENT expectation, and confirm the mock isn't executing real code. (Illustrative: A partial mock stubbed one method on a client but left another untouched; the untouched method fell through to the real implementation and made a live call during the test run.)
• Secret-shaped fixtures must not ship the flagged bytes. Code that handles secrets needs fixtures shaped like real secrets — but a scanner-recognized token written as a contiguous literal trips upstream secret scanners (push protection, pre-commit hooks) and blocks the push, even though the value is fake. Build the token at runtime by splitting the provider prefix across adjacent literals ("xox""b-…"), so the file's bytes never contain the flagged prefix while the runtime value keeps the real shape; grep the source for the contiguous prefix before committing. (A secret scrubber's test fixtures held crafted provider-prefixed tokens as plain literals; GitHub push protection matched its own secret signatures and rejected the push until the prefixes were split across adjacent strings.)

Pre-flight check — before you trust a fixture-backed test

• Assertions target the specific rows/records the test created, not aggregates over a shared store.
• Every foreign key / reference points to a real parent the test created, using its returned id.
• Randomness is seeded and time is pinned, so fixtures are deterministic.
• Any guard test's violating fixture actually trips the failure path.
• Mocks don't secretly load the real module, and no assertion is satisfied by echoing the input.
• No secret-shaped fixture commits a scanner-recognized token as a contiguous literal — the flagged prefix is split and rebuilt at runtime (verified by grep).

If any box is unchecked, the fixture doesn't mirror reality — go make it real before you trust the green.