Properties

Properties: tests that assert what is always true

An example test checks one input. A property test states something true of all inputs and lets a generator hunt for a counterexample.

The tool is not the hard part. hypothesis is mature, installed everywhere, and takes one decorator. The hard part is the blank page after it:

What is actually true about this function, for every input?

Most people freeze there, write @given(st.integers()) with a trivial assertion, and conclude property testing is overrated. It is not — they were missing a catalog. There is a known, small set of property shapes that covers most real code, and applying it is mechanical once known.

Step 1: find the property by working the catalog

Do not stare at the function hoping for inspiration. Go through the patterns and ask which apply. Usually two or three do.

references/catalog.md has a worked example of each, and the questions that reveal which one fits.

The single highest-yield pattern is oracle. If the standard library, an older implementation, or a deliberately naive version computes the same thing, the property is "they agree" — and it is nearly free to write.

Step 2: write the property, not a restatement of the code

The failure mode that ruins property testing is re-implementing the function inside the test. The test then shares the function's bug and passes forever.

# Worthless -- the test computes the same thing the same way.
@given(st.lists(st.integers()))
def test_total(xs):
    assert total(xs) == sum(x * RATE for x in xs)

# Useful -- states a relation that does not depend on the formula.
@given(st.lists(st.integers(min_value=0)))
def test_total_is_monotonic(xs):
    assert total(xs + [1]) >= total(xs)

A property should be shorter and dumber than the implementation. If it is as complicated, it is a second implementation, not a specification.

Step 3: measure whether the property has teeth

A passing property test proves nothing on its own. assert sorted(xs) == sorted(xs) passes for every input ever generated and catches no bug.

scripts/property_strength.py --impl src/rle.py \
    --property-command 'pytest -q tests/test_rle.py::test_roundtrip' \
    --suite-command    'pytest -q'

It breaks the implementation one mutation at a time and sorts each into:

PROPERTY     the property failed         -> what the property buys you
suite-only   only other tests failed     -> the property adds nothing here
NOBODY       everything passed           -> a real gap

A tautology scores near zero. A good property scores high and catches things the example tests miss.

Step 4: when strength is low, diagnose which half is weak

This is the step that matters, because there are two very different causes and they look identical from the outside.

The property is weak — it does not constrain enough. Strengthen the assertion or pick a different pattern from the catalog.

The generator never produces the interesting input — the property is correct but was never fed a case that would expose the bug. This is common and easy to miss.

A real example: for run-length encoding, decode(encode(s)) == s under @given(st.text()) scores 86%. The surviving mutant is the counter increment. st.text() draws from all of unicode, so it almost never produces adjacent repeated characters — the only inputs where run-length encoding does anything at all. One change:

@given(st.text(alphabet="ab"))   # repeats become common

Same property, same assertion. Strength goes to 100%.

Narrower alphabets and smaller value ranges usually find more bugs, because collisions, duplicates, and boundaries become likely instead of astronomically rare. When strength is low, suspect the strategy before rewriting the property.

Step 5: keep the counterexample

When a property fails, hypothesis shrinks the failure to a minimal input. Record it as a permanent example test alongside the property:

def test_regression_empty_run():
    assert decode(encode("aa")) == "aa"   # found by test_roundtrip, 2026-07

The property guards the general case; the example pins the specific bug so it cannot come back silently if a strategy changes.

What not to do

• Do not chase strength as a score. It is a diagnostic. A property at 70% that states something meaningful beats a contrived one at 100%.
• Do not write a property to kill a specific mutant. Same failure as writing a test to kill a mutant — it ends up asserting an implementation detail.
• Do not delete example tests. Properties and examples do different jobs; examples document intent and pin known bugs.
• Do not add @settings(max_examples=10000) to force a pass. If a property only fails rarely, the generator is wrong, not the budget.

Resources

• scripts/property_strength.py — measures what a property actually catches, split into property / suite-only / nobody. Reuses the mutation engine from the mutants skill in this plugin.
• references/catalog.md — every pattern with a worked example, the questions that reveal which fits, strategy design, and the anti-patterns.

Related: mutants finds the gaps in a suite; this skill fills them. A survivor on a comparison operator is almost always best answered with a property rather than another example.