# Pytest Optimizer

> Use when a pytest suite or its fixtures are slow and need measured, safety-gated optimization across the full pipeline.

- Skill: `tony/pytest-optimizer` (Agent Skill, multi-file: 13 files)
- Install (CLI): `npx skillmds@latest add tony/pytest-optimizer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tony/pytest-optimizer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tony (https://skillmd.com/u/tony)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tony/pytest-optimizer

---


# pytest-optimizer

Profile a pytest suite, find **safe** speedups, and apply each one as its own
verified commit — through a resumable, idempotent four-phase loop.

The core discipline: **measure before you mutate, and prove a speedup against the
project's own timing noise before you trust it.** A theoretically great change
whose measured delta falls inside the noise band is worth nothing and is never
committed.

`$ARGUMENTS` may name a test path or marker to scope the run, a `--phase` to jump
to, `--force` to ignore idempotency tokens and re-run, and `--memory-dir` to
override where durable state is written.

## The loop

```
00-scan  ->  01-benchmark  ->  02-plan  ->  03-execute  ->  (re-scan)
```

Each phase is a separate command that reads and writes durable JSON under the
resolved memory directory. Re-running the whole skill resumes at the first
incomplete phase. After `03-execute`, re-scanning measures the new baseline and
the loop can continue until no candidate clears the rubric.

| Phase | Command | Reads | Writes |
|-------|---------|-------|--------|
| 1 | the `pytest-optimizer-00-scan` skill | `AGENTS.md`/`CLAUDE.md`, prior `state.json` | `baseline.json`, `capabilities.json`, `hypotheses.json` |
| 2 | the `pytest-optimizer-01-benchmark` skill | `baseline.json`, `hypotheses.json` | `benchmarks.json` |
| 3 | the `pytest-optimizer-02-plan` skill | `benchmarks.json`, `baseline.json` | `plan.json` |
| 4 | the `pytest-optimizer-03-execute` skill | `plan.json`, `baseline.json` | `execution-log.json` + commits |

## Step 1: Detect preferred tools and the project test command

Check for the user's preferred search tools, then read the project's own test and
quality-check commands. **Never hardcode the runner** — `pytest`, `uv run`, `tox`,
`nox`, and `hatch` are all valid and the project declares which it uses.

```bash
for tool in rg ag fd jq uv uvx; do command -v "$tool" >/dev/null 2>&1 && echo "$tool:available" || echo "$tool:missing"; done
```

Read `AGENTS.md`, `CLAUDE.md`, `justfile`, `tox.ini`, and `pyproject.toml` to
discover the canonical commands for running the suite, the formatter, the linter,
and the type checker. Record the test command verbatim; every phase reuses it.

## Step 2: Detect pytest capabilities (version gating)

The toolbox depends on the installed pytest and plugins. Probe once and persist
the result, so later phases use only features that exist.

```bash
python -c "import pytest; print(pytest.__version__)"
```

```bash
pytest --help 2>/dev/null | rg -- '--durations-min|--import-mode|--dist|--sw-reset' || true
```

Resolve each feature against `references/version-matrix.md`
(min version, detection probe, and graceful fallback). For example: pre-6.1 pytest
has no `--durations-min` (post-filter rows yourself); `--dist=loadgroup` needs
`pytest-xdist >= 2.5`; the `Stash` API needs pytest >= 7.0 (else use
`config.cache`). Write the resolved matrix to `capabilities.json`.

## Step 3: Resolve the memory directory

Follow `references/memory-schema.md`. Prefer
`.pytest-optimizer/` at the repo root and add it to `.gitignore` idempotently. If
the tree is read-only or the user declines an in-tree directory, fall back to the
XDG cache path. Record the chosen path in `state.json` so all four phases agree.

## Orchestration Plan

Before any measurement or edit, present a plan and get approval.

1. **Enter plan mode.** In Claude Code call `EnterPlanMode`. In Cursor/Codex/Gemini
   use `/plan` or `Shift+Tab`. If plan mode is unavailable, the phase structure
   below still forces analysis before any write.
2. **The orchestration plan must state:**
   - the resolved test command and quality-check commands (from Step 1);
   - the detected pytest + plugin versions and which capabilities are gated off
     (from Step 2);
   - the resolved memory directory and whether `.gitignore` will be touched;
   - which phase the run will start at (fresh `00-scan`, or resume), and the
     scope (`$ARGUMENTS` path/marker, or the whole suite);
   - that `00-scan` and `02-plan` only read and write JSON memory — no suite edits —
     and that suite edits happen only in `03-execute`, one commit per speedup,
     each green-verified.
3. **Present the plan and wait for approval.** Let the user narrow scope, exclude
   risky heuristics, or cap the number of commits.
4. **Exit plan mode** before running any phase.

## Step 4: Run the phases

Route to the four commands in order, honoring `--phase` and resume state. Each
command embeds its own detailed procedure; this skill only sequences them and
carries the shared context (test command, capabilities, memory dir).

- **`00-scan`** establishes the baseline (median + MAD noise band over N serial,
  cache-disabled runs), profiles slowest tests (`call` rows) and fixture
  setup/teardown (`setup`/`teardown` rows), and runs the static detectors. It
  emits `hypotheses.json`: candidate speedups, each tagged with a heuristic id
  (`H01`–`H44`) and the goal it addresses.
- **`01-benchmark`** applies each hypothesis in isolation on a throwaway working
  copy, re-measures vs the noise band, and runs the safety gates (order
  independence, collection determinism, green re-run). Within-noise or
  gate-failing candidates are rejected with a reason.
- **`02-plan`** scores the survivors with the rubric
  (`references/scoring-rubric.md`), drops anything below the
  safety gate, orders the rest (typing and safety-gate fixes first; scope and
  consolidation before parallelism), and drafts one commit per speedup.
- **`03-execute`** applies the plan one item at a time: edit, run the project
  quality checks + suite, commit alone on green (why/what message), revert and
  skip on failure. It checkpoints after every item and resumes from the last
  applied index.

## Step 5: Report and decide whether to loop again

After `03-execute`, re-measure total wall-time against `baseline.json` and present
the result using the shared output contract
(`references/output-contract.md`): a hero block, the
applied/skipped speedups with their measured deltas, and an `ask-user-choice`
next-step panel (re-scan for a second pass, scaffold the opt-in cache/plugin, or
stop).

## Opt-in scaffolds

When profiling shows it would help, offer (never impose) to generate a
project-owned, gitignored helper:

- A `pytest11` timing plugin (`templates/pytest_optimizer_plugin.py`) that wraps
  `pytest_fixture_setup` to attribute setup cost to the **individual** fixture —
  the one thing pytest core cannot do, since a `setup` duration row is the whole
  setup chain for an item.
- A `config.cache` memoization snippet (`templates/conftest_cache_snippet.py`)
  with an invalidation token and an xdist write-guard, for expensive build
  artifacts recomputed every run.

Generate only on explicit confirmation; gitignore the output unless the user
chooses to commit it.

## Graceful degradation

- **No plan mode** — keep the phase order; never touch the working tree during
  `00-scan`, `01-benchmark`, or `02-plan` (`01-benchmark` measures only on an
  isolated throwaway copy).
- **Old pytest** — gate features via the version matrix; widen the noise band when
  duration precision is coarser (pre-6.0 used `time.time()`, not `perf_counter`).
- **No xdist / randomly / deadfixtures** — fall back to serial runs, manual
  reversed-order isolation checks, and the built-in `--fixtures` vs
  `--fixtures-per-test` diff.
- **Dirty working tree** — refuse to run `03-execute` until clean (one commit per
  speedup requires a clean base); `00-scan` and `01-benchmark` leave the working
  tree untouched (`01-benchmark` works on an isolated copy) and may run anyway.

## References

- `references/scoring-rubric.md` — the weighted rubric and
  the hard safety gate.
- `references/heuristic-catalog.md` — `H01`–`H44` with
  detect/action/risk/effort and the goal each addresses.
- `references/version-matrix.md` — feature → min version →
  probe → fallback.
- `references/memory-schema.md` — memory location, file set,
  idempotency token, baseline-diff.
- `references/durations-parsing.md` — invoking and parsing
  `--durations`, phase bucketing, JUnit XML, the noise band.
- `references/fixture-analysis.md` — scope model and the
  unused/duplicate/mis-scope recipes.
- `references/safety-gates.md` — order independence,
  collection determinism, xdist preconditions.
- `references/parametrize-convention.md` — the typed
  NamedTuple + test_id convention and typed builtins.
- `references/output-contract.md` — the shared output shape.


## Portability notes

- `ask-user-choice` — present the listed options and wait for the user to pick one. Hosts with a structured multiple-choice tool (Claude Code's `AskUserQuestion`) should use it; otherwise print a numbered list and wait for a numbered reply. Never proceed on an assumed answer.
- `$ARGUMENTS` — the text the user passed when invoking this skill. If your host does not substitute it, read it as the user's request in the current turn, and ask when there is none.
- Bundled files — every relative path in this skill points at a file shipped inside this skill directory. Read them from here, not from the host's plugin tree.