Testprof

TestProf: RSpec Performance Optimization

Expert guidance for diagnosing and fixing slow RSpec test suites. Based on field-tested techniques from Evil Martians' work reducing a 30,000-spec Rails suite from 4m30s to 2m (and targeting 1m30s with parallelization).

The core idea: slow test suites are rarely slow for one reason. They're slow for a handful of fixable reasons, stacked. This skill walks you through diagnosing which reasons apply, then fixing each one — in an order that avoids wasted effort.

How to use this skill

When a user brings a slow RSpec suite, investigate their actual codebase before prescribing anything. A skill full of generic advice is worse than useless — the value is in mapping the general patterns onto their specific code. Follow this order:

1. Characterize the problem — ask for total runtime, example count, and whether it's "slow everywhere" or "slow in specific files." This determines which profiler to run first.
2. Investigate concretely, don't speculate. Before giving advice, actually:
   • Read spec/spec_helper.rb and spec/rails_helper.rb for suspicious requires (rubocop/rspec/support, SimpleCov config, global hooks).
   • Grep the Gemfile for the suspects: SimpleCov, blind_index, any encryption gem, retry gems (retriable, Faraday retry middleware).
   • For slow individual specs, read the service file being tested — grep for sleep, retry, ObjectSpace, create_list(:..., with large numbers.
   • Look at the factory definitions for the types involved. Association fan-out is a common cause. This is the difference between a suggestion list and a diagnosis. Always do the investigation first.
3. Run the right diagnostic next — see "Diagnostic decision tree" below. If Claude can't run the profiler (no shell access to the user's env), write the exact commands the user should run and explain how to interpret the output.
4. Apply fixes in order of payoff — coverage and GC issues often give the biggest single wins and require the least invasive changes, so check them before rewriting specs.
5. Measure after each change — keep the before/after numbers so you can tell what worked.

The four findings in references/bottlenecks.md cover the most common heavy hitters. Read that file when you're ready to dig into a specific bottleneck.

A worked investigation looks like this

Bad (generic suggestion list):

"Your service might be sleeping in tests. You could inject a delay parameter, or stub sleep, or use Retriable with base_interval: 0..."

Good (grounded in the user's code):

"I read app/services/your_service.rb — the retry is hand-rolled at line 42 (sleep(2 ** attempt)), not delegated to a gem. So injection is the right fix, not gem config. Here's the diff..."

The second version took one Read call and one Grep but produced a fix the user can apply immediately. Always aim for the second version.

Diagnostic decision tree

Use the symptoms the user describes to pick the first profiler. Don't try to run all of them — start with the one most likely to surface signal, then pivot based on what you learn.

Rule of thumb: if the user hasn't profiled yet, start with StackProf and a TPS run. Global issues and slow-file issues are usually independent problems.

Quick install

TestProf v1.6+ has all the profilers referenced here:

# Gemfile
group :test do
  gem "test-prof", "~> 1.6"
end

Most profilers activate via environment variables — no code changes needed. For let_it_be, before_all, and FactoryProf, add the relevant requires to spec/spec_helper.rb (see references/testprof_features.md).

The four big wins

From a real 30k-spec suite, these accounted for the majority of the speedup. Any of them could apply to the user's suite. See references/bottlenecks.md for the full diagnostic steps and code for each.

1. RuboCop rubocop/rspec/support causing GC bloat — global hooks keep memory allocated across every spec. Replace with narrower requires. (~1 minute saved on the reference suite.)
2. Coverage tracking overhead (15-20%) — switch SimpleCov to oneshot_line mode. (~1 minute saved.)
3. Encryption/blind_index using production parameters in tests — override to pbkdf2_sha256 with cost: 1 in the test environment. (~30s saved.)
4. Individual slow specs found via TPS — patterns include ObjectSpace iteration, production retry backoffs, excessive record creation for boundary tests. Each fix is local but compound.

When to reach for let_it_be vs let

let_it_be (from TestProf) memoizes across an entire describe block instead of per-example. For read-only records used across many examples, this is a massive win:

# Before: creates a user for every example
let(:user) { create(:user) }

# After: creates once for the whole describe block
let_it_be(:user) { create(:user) }

Use let_it_be when the record doesn't need to be fresh per-example. If examples mutate it, either keep let or use let_it_be(:user, reload: true) / refind: true.

See references/testprof_features.md for the full set of TestProf primitives (before_all, AnyFixture, FactoryProf, EventProf, sample modes).

Tests-per-second (TPS): the metric that matters

Total runtime hides the real problem. A file with 1,080 examples that runs in 2m50s is 6 TPS — terrible. A file with 5 examples that runs in 10s is 0.5 TPS — worse per-example, but maybe unavoidable.

TPS prioritizes files where high runtime meets high example count. That's where refactoring pays off most:

# Show the 20 slowest files with at least 5 examples
TPS_PROF=20 TPS_PROF_MIN_EXAMPLES=5 bin/rspec

When a file shows up in TPS, read references/slow_spec_patterns.md for the common root causes (ObjectSpace walks, sleep/retry delays, oversized fixtures).

Preventing regressions

After optimizing, the suite will grow. To keep gains:

• Enforce in review: code review (human or AI) should flag let that could be let_it_be, tests that create hundreds of records for pagination/boundary checks, and reintroduction of require "rubocop/rspec/support".
• Track TPS over time: the test-queue parallelizer plus periodic TPS runs will surface regressions.
• CI budget: pick a target runtime and fail CI when it slips.

References

Read these when you need more depth on a specific topic:

• references/bottlenecks.md — the four major bottlenecks with full diagnosis steps, code, and expected impact
• references/testprof_features.md — complete TestProf v1.6 feature reference (let_it_be, before_all, profilers, AnyFixture, FactoryProf, EventProf)
• references/slow_spec_patterns.md — common root causes inside individual slow specs, with before/after code
• references/diagnostic_scripts.md — drop-in snippets: GC tracker, timing decorator, suite summary hook