Python Testing

python-testing

Build test infrastructure for a Python package that stays trustworthy under pressure: pytest configuration the installed pytest actually reads, branch coverage with a gate proven to fail, Hypothesis and mutation testing as backstops against tests that execute code without checking it, and multi-version runs. The failure this skill prevents is green-but-meaningless — gates that silently pass, config tables that are silently ignored, and suites edited (by humans or agents) until they stop proving anything.

When NOT to use

• Writing tests for a single file or function, or improving one module's tests — do that directly; no infrastructure needed.
• Diagnosing or fixing one failing test — read the test and the code under test.
• CI workflow YAML — GitHub Actions jobs, version-matrix wiring, coverage upload/combine across CI legs. The python-ci skill, if installed, owns that. This skill decides what runs; CI decides where.
• Packaging work — pyproject metadata, build backend, converting a repo to src layout as a packaging exercise. The python-packaging skill, if installed.
• Git hook wiring — the python-precommit skill; full test suites do not belong in pre-commit hooks anyway.

Workflow

1. Survey the repo

Run the read-only audit first — it catches shadowed and split config sources:

python3 "${CLAUDE_SKILL_DIR}/scripts/check_test_config.py" --root .

Then establish the facts later steps depend on:

uv run pytest --version            # major version decides the config table (step 2)
grep -n "requires-python" pyproject.toml
ls pytest.ini setup.cfg tox.ini .coveragerc noxfile.py 2>/dev/null

Note the layout: src/<pkg>/ + sibling tests/ (src layout) or package at the repo root (flat). Convention throughout: invoke tools through uv (uv run, uvx). If uv is unavailable, the plain fallback is python -m pip install -e ".[dev]" then python -m pytest — same flags everywhere; stated once here.

2. pytest config — one table, in pyproject.toml

Two tables exist, and the installed major decides: pytest >= 9.0 (Nov 2025) reads a native [tool.pytest] table (real TOML arrays/booleans); every supported pytest reads [tool.pytest.ini_options] (INI-style strings). Never write both. On pytest below 9, [tool.pytest] is ignored silently: no error, tests still run, strictness flags never apply — so check uv run pytest --version first (https://docs.pytest.org/en/stable/reference/customize.html). The block below uses ini_options because it works everywhere; on a >= 9 repo prefer the native table with the same keys.

[tool.pytest.ini_options]
minversion = "8.0"          # raise to "9.1" if you rely on the strict flags below via addopts (see gotchas)
testpaths = ["tests"]
addopts = "-ra --strict-markers --strict-config"
filterwarnings = ["error"]

• --strict-markers — typo'd or undeclared @pytest.mark.* markers fail instead of silently never running.
• --strict-config — typo'd config keys become errors.
• filterwarnings = ["error"] — deprecations fail now, not at upgrade time. On a legacy dependency tree this is noisy; add targeted "ignore:<msg>:<category>" entries rather than deleting the gate.
• Async project? Add asyncio_mode = "auto" (with pytest-asyncio as a dev dep) to drop the per-test marker.
• Keep --cov OUT of addopts. Coupling every run to coverage slows single-test loops and can confuse debugger breakpoints; pass coverage flags in the explicit commands below and in CI. Trade-off: some teams keep it in addopts so coverage is never forgotten — if the user prefers that, keep it and note --no-cov disables it for one run.
• One config source. pytest uses the first of pytest.toml/.pytest.toml, pytest.ini/.pytest.ini (both win even when empty), pyproject.toml (with either table), tox.ini, setup.cfg and ignores the rest — a stray pytest.ini silently shadows everything in pyproject. Merge, then delete.

3. Layout and discovery

src layout (src/<pkg>/ with tests in a sibling tests/, not inside the package) is the reliable shape: with uv run, pytest imports the installed package, so import and packaging bugs surface in tests instead of in users' environments (https://docs.pytest.org/en/stable/explanation/goodpractices.html). testpaths = ["tests"] and coverage source = ["src"] then line up.

• "pytest can't find/import my package" right after moving to src layout usually means the package is not installed in the environment — uv sync (or pip install -e .) and re-run.
• Do not migrate a flat-layout repo to src layout just for tests; that is a packaging change (python-packaging territory). Testing works in flat layout — you only lose the installed-package guarantee.
• Shared fixtures live in tests/conftest.py.

4. Coverage — branch on, gate proven

[tool.coverage.run]
source = ["src"]
branch = true
parallel = true
relative_files = true

[tool.coverage.report]
fail_under = 85
show_missing = true
skip_covered = true
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
    "if __name__ == .__main__.:",
]

• branch = true is the point — line coverage says a line ran; branch coverage says both sides of each if did. Line-only numbers overstate.
• fail_under of 80–90 is the defensible band. 100% is anti-recommended: it breeds assert-free tests and pragma sprawl. Exclude genuinely unreachable lines instead.
• If a .coveragerc exists it silently wins over [tool.coverage.*] in pyproject — keep exactly one source.
• Run with uv run pytest --cov --cov-report=term-missing.

Prove the gate. pytest-cov below 7.1 could return exit code 0 on a missed aggregate threshold depending on report options (fixed in 7.1.0, 2026-03), and rounding once passed 66.6% against a 67 bar. Never assume — demonstrate:

uv run pytest --cov --cov-fail-under=100 -q; echo "exit=$?"

Expect a non-zero exit (unless coverage is genuinely 100%). If it exits 0, gate in two steps instead — uv run pytest --cov then uv run coverage report --fail-under=85 — the standalone coverage command's exit code is reliable.

• Parallel runs (pytest -n auto via pytest-xdist) require parallel = true and relative_files = true. Subprocesses spawned by tests need [tool.coverage.run] patch = ["subprocess"] (coverage >= 7.10) — pytest-cov 7 dropped its own subprocess measurement — otherwise coverage comes back empty or 0%.
• A threshold is a floor, not a quality signal — it is trivially satisfied by tests without assertions. Steps 5–6 are the countermeasures. Patch/delta coverage on PRs is CI wiring (python-ci skill, if installed).

Flag-by-flag detail, precedence rules, and combine mechanics: references/configuration.md.

5. Property-based tests where they pay

Add Hypothesis for parsers, serializers, codecs, and algorithmic code — anywhere a property (round-trip, idempotence, invariant) exists. A 426-project corpus study found each property-based test kills roughly 50x the mutations of an average unit test (https://dl.acm.org/doi/10.1145/3764068).

uv add --dev hypothesis    # exact version pinned by uv.lock

from hypothesis import given, strategies as st

@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s

• Prefer bounded strategies (st.integers(min_value=1)) over assume() filtering — rejecting most examples raises HealthCheck.filter_too_much.
• Register settings profiles in tests/conftest.py; CI runs the heavy one via pytest --hypothesis-profile=ci:

from hypothesis import settings
settings.register_profile("ci", max_examples=1000)
settings.register_profile("dev", max_examples=25)

• Persist the .hypothesis/ example database (commit it, or cache it in CI) — otherwise every run rediscovers known counterexamples from scratch.
• DeadlineExceeded only on CI machines means slow shared runners — raise or disable deadline in the ci profile deliberately; don't delete the test.

Property patterns, strategy design, and stateful testing: references/beyond-coverage.md.

6. Mutation testing — do the tests notice bugs?

Coverage proves a line executed; mutation testing proves a test fails when that line is wrong. It is the direct check on assertion quality, which matters most when agents write or edit tests: a suite that was edited until green proves nothing by itself. Options — no settled winner, choose per constraints:

Install as a dev dependency so it runs inside the project environment and gets pinned by uv.lock (an unpinned uvx one-off executes whatever was published last — a reproducibility and supply-chain hole):

uv add --dev mutmut
uv run mutmut run

• Scope the first run to one core module ([tool.mutmut] source_paths; do_not_mutate for generated code), then triage survivors: a surviving mutant means a missing or weak assertion — or a genuinely equivalent mutant, which you mark and move on.
• Do not gate CI on mutation score on day one; run it scoped or nightly and treat the score as a diagnostic — mutmut 3.7.0's own unreleased notes still list coverage-excluded lines being mutated, false survivors and timeouts counted as kills. Practitioners report 85–95% mutation scores as far more meaningful than the same number in line coverage.

Triage workflow and the agent-era rationale: references/beyond-coverage.md.

7. Multi-version runs

Match the tool to the project — this debate is genuinely unsettled:

for v in 3.11 3.12 3.13 3.14; do   # the versions requires-python promises; 3.10 is EOL 2026-10
  uv run -p "$v" --with pytest --with pytest-cov pytest || exit 1
done

For tox or nox, swap the environment backend to uv (tox-uv, nox's uv backend): environment setup gets dramatically faster with no config-model change.

• nox footgun: a repo-root .python-version file can silently pin every session to the local dev version (https://github.com/wntrblm/nox/issues/1038).
• Never enforce fail_under inside each version leg — a 3.13-only branch drops coverage on the 3.10 leg. Locally: run the loop with parallel = true, then uv run coverage combine && uv run coverage report --fail-under=85 once. The CI version of that split (artifacts, combine job) belongs to python-ci.

Configs and trade-offs in depth: references/multi-version.md.

8. Verify

1. uv run pytest — green.
2. The gate-trip demonstration from step 4 exits non-zero.
3. python3 "${CLAUDE_SKILL_DIR}/scripts/check_test_config.py" --root . exits 0.
4. Report to the user: config location and table chosen (and why), thresholds, and which backstops (Hypothesis, mutation, multi-version) are wired vs deferred.

Output spec

Done means:

• Exactly one pytest config source and one coverage config source; the pytest table matches the installed major version.
• Branch coverage on; fail_under in the 80–90 band; the gate demonstrated to exit non-zero when missed.
• Suite runs green via uv run pytest from a clean checkout.
• Where requested: Hypothesis dev dep with profiles; mutation tool pinned with a first scoped run triaged; multi-version runner chosen with stated rationale.
• No CI YAML written or modified.

Failure modes & gotchas

Files

• scripts/check_test_config.py — read-only audit of pytest/coverage config (shadowed sources, wrong config table, missing branch coverage or gate). Exits non-zero when it finds errors; never modifies the repo.
• references/configuration.md — pytest and coverage config in depth: precedence, the pytest-9 table question, the exit-code bug, xdist/combine mechanics.
• references/beyond-coverage.md — Hypothesis patterns and settings; mutation tool comparison and survivor triage; why agent-edited green suites need backstops.
• references/multi-version.md — uv loop vs tox vs nox, uv-backed runners, coverage across versions.