Optimize Test Suite

Optimize Test Suite

Reduce suite runtime and maintenance cost without weakening its ability to catch regressions. Measure first, make one evidence-backed optimization at a time, and compare against the baseline.

Establish the Contract

1. Read applicable repository instructions and test-running guidance.
2. Identify supported interpreters, environments, test commands, markers, shards, and integration gates. Do not conflate environments.
3. Record the requested scope and success criteria: wall time, CPU time, collection time, slowest tests, flake rate, or maintenance duplication.
4. Preserve externally visible behavior, safety invariants, failure semantics, and required test categories.
5. Do not add a dependency, enable uncontrolled parallelism, or change CI policy unless explicitly requested.

Measure a Baseline

Run the smallest representative command that exposes the problem, then the authoritative suite when practical.

Capture:

• exact command and environment;
• collected, passed, skipped, failed, and deselected counts;
• wall-clock duration and framework timing output;
• slowest tests and setup/call/teardown phases when supported;
• repeated-run variance when timings are noisy;
• whether caches are cold or warm.

Use existing profiler, duration, trace, or timing features before writing instrumentation. Do not optimize from intuition alone.

Classify the Cost

Assign each candidate one primary cause:

• duplicated scenario or equivalent assertion;
• repeated expensive fixture or application bootstrap;
• unnecessarily broad integration/e2e coverage;
• filesystem, database, network, subprocess, sleep, retry, or polling cost;
• oversized parameter matrix with equivalent partitions;
• repeated data generation, parsing, imports, compilation, or discovery;
• serial execution forced by avoidable shared mutable state;
• timeout masking a hang, race, or production defect;
• test framework or environment misconfiguration.

Separate slow tests from flaky tests. A speed change that hides a race, suppresses a failure, or only raises a timeout is not an optimization.

Prove Redundancy Before Deleting

Compare setup, stimulus, assertion, branch, and failure mode. Treat tests as redundant only when another retained test detects the same meaningful regression under the same relevant boundary.

Keep distinct tests when they protect different:

• public contracts or entrypoints;
• failure paths, boundary values, security rules, or state transitions;
• interpreters, platforms, dependency variants, or integration seams;
• regression histories that remain plausible.

Prefer deleting an exact duplicate. Use parametrization only when cases share one behavior and failures remain easy to identify. Do not compress unrelated behaviors into a clever table merely to reduce line count.

When existing coverage or mutation tooling is available, use it as supporting evidence, not as the sole proof of equivalence. Do not add such tooling without authorization.

Optimize in Risk Order

Apply the smallest effective change, generally in this order:

1. Delete proven duplicate tests, assertions, setup, and obsolete compatibility cases.
2. Reuse existing helpers and fixtures; remove repeated construction.
3. Narrow setup to the data and subsystem the test actually needs.
4. Replace real sleeps with deterministic clocks, events, or injectable polling seams.
5. Replace accidental external I/O with existing fakes at the correct boundary.
6. Move repeated immutable setup to a broader fixture scope only after proving isolation and cleanup.
7. Split fast contract tests from deliberately gated integration/e2e tests without removing either contract.
8. Reduce parameter matrices using behavioral equivalence classes and retain boundary cases.
9. Enable sharding or parallel execution only when isolation is proven and the repository already supports it or the user requests it.

Fix production code only when profiling exposes an actual product defect or a missing testability seam and the task authorizes that change. Do not distort production APIs solely for test speed.

Guard Against False Improvements

Reject changes that improve timing only by:

• skipping, deselecting, xfail-ing, or weakening assertions;
• reducing retries that were exposing a real race without fixing it;
• sharing mutable fixtures across tests;
• relying on order, warm caches, leaked processes, persistent databases, or developer-machine state;
• mocking the behavior under test instead of its external dependency;
• moving slow tests out of the default command without an explicit suite contract;
• comparing different test counts, environments, or cache states without explanation.

Verify

1. Run focused tests after each change.
2. Run the same baseline command with the same environment and comparable cache state.
3. Compare test counts, outcomes, slowest phases, wall time, and variance.
4. Run tests that validate collection, markers, suite boundaries, fixtures, or CI commands when changed.
5. Repeat timing samples when the improvement is near measurement noise.
6. Run the broader authoritative suite when risk and cost justify it.
7. Inspect the diff and confirm every deletion has retained failure-detection evidence.

Restore or revise any change that causes new flakes, order dependence, leaked resources, reduced contract coverage, or insignificant speedup relative to complexity.

Report

Summarize:

• files changed and tests deleted, consolidated, or repartitioned;
• root causes of redundancy or slowness;
• before/after commands, counts, timings, and sample conditions;
• evidence that regression protection remains;
• remaining bottlenecks, measurement limits, and risks.

State measured improvement numerically. Do not claim the whole suite is faster when only a subset was measured.