Property-Based Testing
Example tests check the handful of inputs you thought of. Property-based tests
check a property against thousands the framework generates — and when one fails,
it shrinks the failure to the smallest input that still breaks, handing you a
minimal repro instead of a haystack. Reach for it where the input space is too
big to enumerate and a clear invariant exists.
Where it earns its keep
- Roundtrips:
decode(encode(x)) == x for all x — serializers, parsers,
protobuf/JSON codecs, compression.
- Invariants: a sorted list is sorted and a permutation of the input; a
balance never goes negative; a state machine never reaches an illegal state.
- Equivalence / oracle: the fast/new implementation agrees with the
slow/reference one for all inputs (the parallel-run check in a rewrite).
- Idempotence / commutativity:
f(f(x)) == f(x); apply(a,b) == apply(b,a).
Finding the property
The skill is naming the property, not writing the generator. Ask: what must be
true regardless of the input? Common shapes — "there and back" (roundtrip),
"different path, same result" (oracle/commutativity), "some things never change"
(invariant), "doing it twice = doing it once" (idempotence). If you can't state
one, the code may not have a crisp contract yet — that's itself a finding.
Shape of a test
from hypothesis import given, strategies as st
@given(st.binary())
def test_roundtrip(payload):
assert decode(encode(payload)) == payload # holds for ALL payloads
Stateful histories and read-only observations
For history-bearing stores, generate valid action sequences against a small
reference model: repeated updates, ownership changes, observations and reloads.
Check required state, proofs and unrelated fields after every action; shrink the
sequence, not just its final input.
- Growth: bound encoded bytes, retained-record counts and nesting against
the required retained payload at increasing sequence lengths. Include an
increasing-payload control. If each event should be retained once, repeated
fixed-size changes must not recursively embed earlier histories.
- Observation: when an API promises read-only inspection, assert no store
write, replacement or fsync. Unchanged decoded state or bytes alone can hide
a same-byte rewrite. Check file identity/mtime or spy persistence calls;
account separately for declared cache or telemetry effects.
- Controls: exercise updates alone, transfers alone, interleaved actions,
repeated reads and reloads. Retain legitimate history and required evidence;
a smaller file obtained by losing records is not a passing result.
If stored state is unexpectedly large, inspect metadata and the serialization
and mutation paths first. Reproduce with a tiny owned fixture under event,
byte and time limits before loading, copying or hashing the full store. Keep
any necessary production inspection within existing ownership and resource
bounds; synthetic timings do not measure production latency.
Discipline
- Assert behavior, not the implementation — a property restating the code
proves nothing.
- Save the shrunk counterexample as a regression example test — property
tests find the bug; a pinned example keeps it dead (pairs with
regression-ratchet).
- Constrain generators to the real domain (valid ranges, encodings) so
failures are real bugs, not "we never accept that input anyway."
- Don't replace example tests — keep readable examples for the common path;
add properties for the space you can't enumerate. They're complementary.
1---2name: property-based-testing3description: Use when example tests can't cover the input space — parsers, serializers, encoders, money math, state machines, anything with a roundtrip or invariant. Assert properties that hold for all inputs, let the framework generate cases and shrink failures to a minimal counterexample.4metadata:5 source: https://skillmd.com/skills/selamy-labs/property-based-testing6---78# Property-Based Testing910Example tests check the handful of inputs you thought of. Property-based tests11check a property against thousands the framework generates — and when one fails,12it *shrinks* the failure to the smallest input that still breaks, handing you a13minimal repro instead of a haystack. Reach for it where the input space is too14big to enumerate and a clear invariant exists.1516## Where it earns its keep1718- **Roundtrips**: `decode(encode(x)) == x` for all `x` — serializers, parsers,19 protobuf/JSON codecs, compression.20- **Invariants**: a sorted list is sorted and a permutation of the input; a21 balance never goes negative; a state machine never reaches an illegal state.22- **Equivalence / oracle**: the fast/new implementation agrees with the23 slow/reference one for all inputs (the parallel-run check in a rewrite).24- **Idempotence / commutativity**: `f(f(x)) == f(x)`; `apply(a,b) == apply(b,a)`.2526## Finding the property2728The skill is naming the property, not writing the generator. Ask: what must be29true *regardless* of the input? Common shapes — "there and back" (roundtrip),30"different path, same result" (oracle/commutativity), "some things never change"31(invariant), "doing it twice = doing it once" (idempotence). If you can't state32one, the code may not have a crisp contract yet — that's itself a finding.3334## Shape of a test3536```python37from hypothesis import given, strategies as st3839@given(st.binary())40def test_roundtrip(payload):41 assert decode(encode(payload)) == payload # holds for ALL payloads42```4344## Stateful histories and read-only observations4546For history-bearing stores, generate valid **action sequences** against a small47reference model: repeated updates, ownership changes, observations and reloads.48Check required state, proofs and unrelated fields after every action; shrink the49sequence, not just its final input.5051- **Growth:** bound encoded bytes, retained-record counts and nesting against52 the required retained payload at increasing sequence lengths. Include an53 increasing-payload control. If each event should be retained once, repeated54 fixed-size changes must not recursively embed earlier histories.55- **Observation:** when an API promises read-only inspection, assert no store56 write, replacement or fsync. Unchanged decoded state or bytes alone can hide57 a same-byte rewrite. Check file identity/mtime or spy persistence calls;58 account separately for declared cache or telemetry effects.59- **Controls:** exercise updates alone, transfers alone, interleaved actions,60 repeated reads and reloads. Retain legitimate history and required evidence;61 a smaller file obtained by losing records is not a passing result.6263If stored state is unexpectedly large, inspect metadata and the serialization64and mutation paths first. Reproduce with a tiny owned fixture under event,65byte and time limits before loading, copying or hashing the full store. Keep66any necessary production inspection within existing ownership and resource67bounds; synthetic timings do not measure production latency.6869## Discipline7071- **Assert behavior, not the implementation** — a property restating the code72 proves nothing.73- **Save the shrunk counterexample as a regression example test** — property74 tests find the bug; a pinned example keeps it dead (pairs with75 regression-ratchet).76- **Constrain generators to the real domain** (valid ranges, encodings) so77 failures are real bugs, not "we never accept that input anyway."78- **Don't replace example tests** — keep readable examples for the common path;79 add properties for the space you can't enumerate. They're complementary.