Testing Python Libraries

Python Library Testing

Quick Start

uv run pytest                       # Run tests
uv run pytest --cov=my_library      # With coverage
uv run pytest -x                    # Stop on first failure
uv run pytest -k "test_encode"      # Run matching tests

Pytest Configuration

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra -q --cov=my_library --cov-fail-under=85"

[tool.coverage.run]
branch = true
source = ["src/my_library"]

Test Structure

tests/
├── conftest.py           # Shared fixtures
├── test_encoding.py
└── test_decoding.py

Essential Patterns

Basic test:

def test_encode_valid_input():
    result = encode(37.7749, -122.4194)
    assert isinstance(result, str)
    assert len(result) == 12

Parametrization:

@pytest.mark.parametrize("lat,lon,expected", [
    (37.7749, -122.4194, "9q8yy"),
    (40.7128, -74.0060, "dr5ru"),
])
def test_known_values(lat, lon, expected):
    assert encode(lat, lon, precision=5) == expected

Fixtures:

@pytest.fixture
def sample_data():
    return [(37.7749, -122.4194), (40.7128, -74.0060)]

def test_batch(sample_data):
    results = batch_encode(sample_data)
    assert len(results) == 2

Mocking:

def test_api_call(mocker):
    mocker.patch("my_lib.client.fetch", return_value={"data": []})
    result = my_lib.get_data()
    assert result == []

Exception testing:

def test_invalid_raises():
    with pytest.raises(ValueError, match="latitude"):
        encode(91.0, 0.0)

For detailed patterns, see:

• FIXTURES.md - Advanced fixture patterns
• HYPOTHESIS.md - Property-based testing
• ../project-setup/CI.md - CI/CD test configuration

Test Principles

Tests That Lie: Avoiding False-Green

A passing suite is worthless if it can't fail when the code is wrong. The most expensive bugs ship under green CI. Audit for these anti-patterns — each is a way "all tests pass" can mask broken behavior.

Conditional assertions that vacuously pass. If the setup silently fails, a guarded assertion never runs and the test still passes.

# BAD — if the Sphinx build fails, no index.html, so nothing is asserted.
def test_build_includes_css():
    if index_html.exists():            # build broke? test passes anyway.
        assert "theme.css" in index_html.read_text()

# GOOD — assert the precondition, then the behavior.
def test_build_includes_css():
    assert index_html.exists(), "build produced no index.html"
    assert "theme.css" in index_html.read_text()

Over-permissive assertions. An or that can't fail, or a substring match so loose it accepts wrong output.

assert result.returncode == 0 or "html_static_path" in result.stderr  # masks real failures
assert "ui" in todo.tags                                              # also matches "build"

Presence assertions are blind to duplicates. assert X in output proves that at least one X exists. It cannot tell you that there are two, or which one a consumer will honor. That is the exact shape of bugs in generated output — HTML tags, config keys, emitted headers — where a framework's base template and your override each emit one and they disagree. Extract every occurrence and compare the whole list.

# BAD — passes with one correct tag, and equally with two conflicting ones.
assert '<link rel="canonical"' in html

# GOOD — pins both the count and the values.
canonicals = re.findall(r'<link rel="canonical" href="([^"]*)"', html)
assert canonicals == ["https://example.com/guide/"]

When two emitters produce the "same" tag they often don't format it identically (one self-closing, one not), so anchor the pattern on the attribute you care about and stop before the tag close — otherwise the duplicate you're hunting slips past your regex and the test looks clean. The same rule applies to anything countable: assert len(rows) == 2 and the row values, not assert rows.

Mocking the thing under test. If you patch _run_command and only assert the argv tokens, the test locks in a command that may not exist — it stays green even after the subcommands or flags it builds are renamed or removed upstream. Mock at the boundary (the subprocess/HTTP call), then assert on the parsed result, not on the arguments you passed in.

Fakes that ignore the parameters being tested. A fake client whose list_items returns all canned rows in one call — ignoring after and pagination — cannot exercise the pagination or incremental-sync logic those parameters drive, so it stays unverified. Make fakes honor the parameters whose handling is the point of the test.

Smoke tests that import the wrong thing. python -c "import server" can print success by resolving an empty server/ package that shadows the real server.py — a broken wheel that still "imports." Assert a real symbol is reachable (from server import main; main), not merely that an import name resolves.

Forgotten mock → silent real network calls. A test missing its httpx_mock fixture hits the live API: slow, flaky, rate-limited, and silently exercising nothing deterministic. Add --disable-socket (pytest-socket) so any unmocked network call fails loudly instead of "passing."

No-op CI gates. Confirm the gate actually runs the tests:

• go test ./... / pytest with zero test files is a green no-op.
• Files excluded via --ignore or pytest.mark.skip "because flaky" often fail deterministically — exclusion hides real breakage, not flakiness.
• Marker filters (-m "not integration") can deselect the only meaningful tests. Reproduce CI's exact marker expression locally before trusting green.

Empty evaluator sets must not mean "all clear." Auditors, policy engines, and validation pipelines often compute a score from the enabled rules. A category filter can accidentally disable every rule; if the scoring code maps results == [] to score = 100, an unsupported request becomes a silent perfect pass. Treat an empty post-filter rule set as a configuration/error state, or keep an aggregate rule enabled when it is the evaluator for every category.

enabled = [rule for rule in rules if rule.supports(requested_categories)]
if not enabled:
    raise NoApplicableRulesError(requested_categories)
results = evaluate(enabled, subject)

Regression tests must assert more than the final score: request each supported category (and representative combinations), assert that at least one rule ran, and use a known failing subject so a no-op path cannot look clean.

result = audit(known_noncompliant_subject, categories=["bias"])
assert result.rules_evaluated > 0
assert result.issues                 # proves filtering did not bypass evaluation
assert result.overall_score < 100

Tests written around a bug. Wrapping a call in try/except RuinError to make it pass documents the bug as acceptable. Assert the correct behavior and let it fail until the bug is fixed (use xfail(strict=True) to track it without red CI).

Prove the Test Can Fail: Mutate the Fix

Every regression test makes an implicit claim — this would have caught the bug. The only way to check the claim is to put the bug back and watch the test go red. Do it once, while the fix is still fresh in your head; it takes a minute and it is the difference between a regression test and a decoration.

# 1. revert the fix (git stash, or hand-edit the guard back to its broken form)
# 2. run ONLY the new test — it must FAIL, and for the right reason
uv run pytest tests/test_paths.py::test_symlink_escape_rejected -q
# 3. restore the fix — it must pass

Mutate each half of a compound guard separately. A fix that validates an input and re-checks the resolved result looks redundant until you revert each half on its own.

def _resolve_baseline(name: str) -> Path:
    if ".." in name or not NAME_RE.fullmatch(name):   # mutation A
        raise ValueError(name)
    path = (BASE_DIR / f"{name}.json").resolve()
    if not path.is_relative_to(BASE_DIR.resolve()):   # mutation B
        raise ValueError(name)
    return path

Reverting A is caught by a ../../etc/passwd test. Reverting B is caught only by a symlink placed inside the approved directory that points outside it — a test most suites don't have. If reverting one half leaves the suite green, you have a test gap, not a redundant check: write the test that pins it.

Know what "red" looks like for the mutation you chose. A reintroduced bug does not always surface as a clean assertion failure.

• Remove a retry/iteration cap and the suite hangs instead of failing. Run the mutated suite under an external timeout — timeout 60 uv run pytest -q with no output is the reproduction. Making the cap merely unreachable (> 10**9) is the honest mutation; the off-by-one (> → >=) fails loudly and is the cheaper one to re-run day to day.
• Drop an await or a try in async code and the runner may die with an unhandled rejection and a worker crash rather than a named test failure. Still red — read the output before concluding your test didn't fire.
• If the mutated run errors during collection, no test ran at all and you have learned nothing about the test.

A hand-written fixture is itself a mutation — of reality. When the same person authors both a parser and every fixture it is tested against, both encode the same assumption, and the suite is green against a guard that cannot fire in production. A regex requiring single spaces between tokens matches hand-typed single-space fixtures forever, while the live document renders those tokens across indented lines. Capture at least one fixture from the real source, commit it, and point the guard's test at that.

Mutate in Both Directions: A Guard Can Also Fire Too Often

A fix that adds a conditional — a warning on incomplete data, a validation check, a flag that suppresses a claim — has two plausible wrong implementations, not one: it never fires (the original bug), or it always fires (over-broad). The regression test you naturally write covers only the first. Mutate both ways.

# The fix: warn when a sub-fetch failed, so a partial total isn't read as complete.
if failed_sources:
    log.warning("counts UNDERCOUNT: no data from %s", ", ".join(failed_sources))

# Mutation 1 — the original bug. Delete the tracking so the list is always empty.
# Mutation 2 — over-broad. Drop the condition, or write `if not sources:`.

The control test is the one that catches mutation 2, and it looks vacuous. A test asserting that a fully successful run emits no warning passes on the pre-fix code by construction, so it reads like it proves nothing and gets cut in review. It is the only test that fails when the warning becomes unconditional. Keep it, and name the pairing explicitly in the PR body so a reviewer doesn't delete half the coverage:

Truthiness is how guards become over-broad. The over-firing mutation is rarely a deliberate edit — it is if not x: where the intent was "this value is absent". None means unmeasurable / the fetch failed; [], 0, 0.0 and "" are legitimate healthy results that a truthiness check silently folds in with it.

# BAD — fires on a resource that genuinely has no referrers, or a real count of 0.
if not referrers:
    warn_incomplete(name)

# GOOD — only the sentinel means "we don't know".
if referrers is None:
    warn_incomplete(name)

A useful tell that you got this wrong: existing fixtures that pass empty collections start emitting the new warning. If adding a guard turns unrelated tests red, read those failures as the over-broad mutation reporting itself rather than as fixtures needing an update.

Attribute each mutation to exactly one test. When a fix has several parts, revert them one at a time and record which test fails for each. "Deleting only the bool short-circuit fails exactly one test" is a checked statement; "the suite covers this" is not. A part whose removal leaves the suite green is a gap.

Pick Fixture Values That Separate the Bug From the Fix

A test survives mutation most often because its data cannot tell the two implementations apart. Before writing the assertion, work out what the buggy version would produce from your fixture. If it produces the same number, the fixture is the problem — change it, not the assertion.

Order-sensitive readings need a fixture whose ends differ. A summary that prints the latest value from an already-sorted frame reads .iloc[0] when the sort is descending and .iloc[-1] when it is ascending; both are one character apart and only one is right. A single-row or flat fixture makes them agree.

# BAD — every row carries the same total, so first and last agree and the
# assertion holds against either reading.
rows = [{"day": "2024-01-01", "cumulative": 5}]

# GOOD — strictly increasing across three days: 2, then 5, then 9. Only the
# correct reading yields 9.
rows = [
    {"day": "2024-01-01", "cumulative": 2},
    {"day": "2024-01-02", "cumulative": 5},
    {"day": "2024-01-03", "cumulative": 9},
]
assert "Final cumulative stars: 9" in captured.messages

For per-group derived columns, the two readings must disagree on every row. A gap/delta/rank column computed per group and the same column computed over the globally sorted table often coincide for most rows; interleaving two groups is not enough on its own — the values have to come out different. Two items per group, timed so the within-group spacing and the global spacing diverge, is the smallest fixture that pins it:

Per group there are two blanks and gaps of 10.0 and 15.0; globally there is one blank and no value matches. Assert the exact numbers and the blank count.

A fixture that reproduces the bug's magic constant proves nothing. When the defect is a redundant fixed wait of 60 seconds and the legitimate code path also falls back to 60 seconds when a header is missing, a fixture that omits the header passes on both. Supply a value that makes the legitimate result distinct — a reset header five seconds out — so the extra wait is visible.

Record the sequence, not the fact. A no-op stub answers "did it sleep?" and nothing else. Collect the arguments instead and assert the whole call sequence; that is what catches one extra call in the middle.

# BAD — passes whether the code sleeps once or twice.
monkeypatch.setattr("mypkg.client.time.sleep", lambda _s: None)

# GOOD — the sequence is the assertion.
sleeps: list[float] = []
monkeypatch.setattr("mypkg.client.time.sleep", sleeps.append)
...
assert sleeps == [reset_wait, 0.1]   # reset wait, then the post-success pause

Changing Behavior an Existing Test Already Asserts

Before you change any output, read what the suite already claims about it. Three situations look identical from a green run:

1. Shape assertions only — isinstance(x, float), >= 0, "minutes" in result. The behavior is unspecified: the test passed before your fix and will pass after it, and would have passed on almost any output. Treat this as no coverage, not as coverage you must preserve.
2. The correct value is pinned. Your "fix" is the regression. Stop.
3. The buggy value is pinned. A confident exact-match assertion on wrong output makes the defect read as tested-and-intended.

Green is not evidence of intent, and the test cannot tell you which of 2 and 3 you are in — only the docs, the docstring, or the issue history can. Both shapes occur in the same file: a renderer that deliberately drops list markers has a test pinning the marker-free output, and the accidental defect two functions over where heading text fuses onto the next block has an equally confident pinned test. If the intent is written down nowhere, writing it down is part of the fix.

Grep for every assertion that pins the old value — not just the one the issue named. A ticket cites the obvious single-case test; the same wrong output is usually also embedded inside a multi-line expected string in some test_mixed_content-style case, where it does not look like the thing you are changing.

# search for a distinctive slice of the OLD output, not for the test's name
rg -l 'HeadingBody text' tests/

Prefer a differential assertion over a magic constant. When the number comes from a third-party computation you do not control — a readability scorer, a tokenizer, a formatter — an exact float pins your test to that library's version. If the dependency is declared unpinned ("textstat", no specifier), a patch release moves the value and the test goes red with no change in your code. Assert the relationship the fix establishes instead:

# BAD — version-brittle, and it only says "some number came out".
assert readability(MARKDOWN)["flesch"] == 42.80

# GOOD — the property the fix creates: markup must not affect the score.
assert readability(MARKDOWN)["flesch"] == readability(PLAIN_EQUIVALENT)["flesch"]
assert strip_markup(MARKDOWN) == PLAIN_EQUIVALENT      # exact, no library constant
assert len(strip_markup(MARKDOWN).split()) == 27

The differential form needs no constant from the dependency at all, and it still goes red the moment the stripper regresses.

If you do pin a number, measure it — never estimate. Derived metrics are not intuitive, and a tolerance tight enough to be meaningful is tight enough that a plausible guess fails on the first CI run. Compute it, then paste the result:

uv run python -c "from mypkg import readability; \
  print(readability(open('tests/data/doc.md').read()))"

Size the tolerance against the bug's own gap rather than your confidence: if the buggy and fixed paths differ by ~29 points, pytest.approx(42.80, abs=1.0) is both a real value assertion and tolerant of a dependency bump.

To quote the "before" number in the test's comment, load the old module directly — you do not need to check out the old tree:

git show main:mypkg/parser.py > /tmp/old_parser.py

spec = importlib.util.spec_from_file_location("old_parser", "/tmp/old_parser.py")
old = importlib.util.module_from_spec(spec)
spec.loader.exec_module(old)
print(readability(old.strip_markup(MARKDOWN)))   # what the buggy path used to yield

Ambient State: Tests That Only Pass on Your Machine

A test that reads state it never set — environment variables, a module-level cache, the working directory, the clock — is testing the machine as much as the code. These are the tests that pass locally and fail in CI, or pass or fail depending on which test ran first. Pin every input the code reads.

Pin every variable that feeds a lookup, not just the one you know about. Config-directory resolution is the classic trap: setting HOME looks sufficient, but on Linux the XDG variables are set independently of HOME, so the lookup ignores your temp dir and every test shares one real config directory. One test's corrupt fixture then leaks into the next, and run order decides who fails.

# BAD — HOME alone. macOS ignores XDG entirely, so this passes locally forever
# and only ever fails on Linux CI.
monkeypatch.setenv("HOME", str(tmp_path))

# GOOD — pin every input to the resolution.
monkeypatch.setenv("HOME", str(tmp_path))
monkeypatch.setenv("XDG_CONFIG_HOME", str(tmp_path / "config"))

When a platform difference decides whether a variable is read, the incomplete version of the test is not "mostly right" — it is untested on exactly the platform that runs CI.

Import-time configuration breaks collection, not tests. A settings object built at module scope (settings = Settings()) is evaluated on import, so a missing variable fails collection — before any fixture runs, so no fixture can fix it. Declare the variables where collection can see them (pytest config, or a root conftest.py), mirroring the env: block CI uses. Better: build config in a factory the test can call with overrides, so importing the module is inert.

Module-level globals outlive the test that populated them. A cache like _analyzers = None keeps one test's mocks alive for every later test. Reset it before and after — the trailing reset is what stops the last test in a file from leaking into the next file.

@pytest.fixture(autouse=True)
def reset_analyzer_cache():
    app._analyzers = None
    yield
    app._analyzers = None

Resetting a mock has the same trap: reset_mock() clears recorded calls but keeps return_value and side_effect, so a stub set in one test still answers in the next.

m.reset_mock()                                       # calls cleared; stub still returns 42
m.reset_mock(return_value=True, side_effect=True)    # actually resets the stub

The working directory is an input. Code that shells out inherits the process CWD. A test asserting "runs against the path I passed" is vacuous when the test's own CWD is already a valid project — it passes whether or not the path is threaded through at all. Move away first, so the fallback would actually fail.

def test_runs_against_given_path(tmp_path, monkeypatch, project):
    monkeypatch.chdir(tmp_path)          # empty: a CWD fallback errors here
    assert not run_tool(project_path=str(project)).startswith("Error")

Freeze the clock and keep it frozen. Restoring the real clock mid-test — to wait on something — silently hands wall-clock time back to any date logic that runs afterward. A business-hours branch then follows whatever the runner's local time happens to be, so the suite is green during the day and red at night. Drive the pending work deterministically instead of sleeping inside a frozen-clock test.

Checklist

Testing:
- [ ] Tests exist for public API
- [ ] Edge cases covered (empty, boundary, error)
- [ ] No external service dependencies (mock them)
- [ ] Each regression test verified red against the reverted fix
- [ ] New guards mutated both ways (never fires / always fires), each mutation
      attributed to one test; control test for the healthy case kept
- [ ] Absence checked with `is None`, not truthiness (`[]`/`0`/`""` are real values)
- [ ] Fixture values make the buggy and correct readings produce different results
- [ ] Existing assertions on changed output classified (shape-only / pins correct
      value / pins the bug), and every file embedding the old value updated
- [ ] Values from third-party computations asserted differentially, or measured
      and pinned with a tolerance sized to the bug's gap — never estimated
- [ ] No ambient state read unpinned (env vars, module globals, CWD, clock)
- [ ] Coverage > 85%
- [ ] Tests run in CI

Learn More

This skill is based on the Code Quality section of the Guide to Developing High-Quality Python Libraries by Will McGinnis. See these posts for deeper coverage:

• Testing with Pytest
• Testing Coverage
• Testing with Tox
• Testing with Mocking