Performance Regression CI

Performance Regression CI

Purpose

A performance gate is a decision system, not a percentage comparison. It must state what change matters, acquire evidence capable of detecting it, distinguish code effects from environment drift, and produce pass, regression, or inconclusive without silently turning missing evidence into success.

The objective is not zero alerts. It is a known operating characteristic: acceptable false block probability, useful power at a declared alternative beyond the decision margin, bounded cost, and an auditable baseline. A green gate with weak power is not evidence of equivalence.

Ownership boundary

This skill owns the automated decision protocol around performance evidence. Delegate:

• benchmark validity, JVM forks, warm-up, profilers, and JMH mechanics to jmh-microbenchmarks and jmh-advanced;
• effect sizes, confidence intervals, quantiles, dependence, and hypothesis design to latency-statistics;
• workload representativeness and coordinated omission to load-testing and coordinated-omission;
• investigation after a signal to performance-methodology.

Gate contract

Write this contract before implementing the comparator:

MPIR is a product/architecture decision, not the observed noise multiplied by a constant. Noise determines whether the available design can resolve that MPIR. If it cannot, improve the design, use a controlled confirmation environment, or admit that the gate is only a coarse screen.

Decision model

Prefer a three-zone decision based on an uncertainty interval for the directional effect:

effect = signed relative change, normalized so positive means worse
M = smallest practically important regression

upper bound < M                 -> no material regression detected
lower bound >= M                -> material regression detected
otherwise                       -> inconclusive
absolute guardrail breached     -> regression, independently of relative baseline

Apply these rules only to valid, comparable measurements. Define whether an absolute guardrail is a deterministic contract violation or an uncertain estimate requiring its own calibrated rule; a broken runner or missing measurement cannot establish a performance breach. With the lower-bound rule, an effect exactly at M is a boundary case, usually inconclusive, not a high-power alternative. Declare the above-M effect at which useful detection power is required.

This is a non-inferiority-style framing. It prevents “not statistically significant” from being translated into “the versions are equivalent.” A team may instead use a calibrated tolerance interval, control chart, Bayesian decision rule, or sequential test; document its assumptions and simulate its error behavior before enabling merge blocking.

For a lower-is-better metric:

signedRelativeChange = current / baseline - 1

For a higher-is-better metric:

signedRelativeChange = baseline / current - 1

The ratio form keeps positive = worse, but it requires positive scores. For zero, negative, or transformed scores, define an absolute effect or a domain-specific transformation. Translate the product margin consistently: a 5% throughput decrease corresponds to 1 / 0.95 - 1, about 5.263% on the inverse-ratio scale, not 5%. Never compare values until benchmark identity, parameter tuple, score unit, mode, and direction are compatible.

Experimental design

Preserve the experimental unit

The independent unit is normally a fresh process/fork or a separately scheduled runner session, not an invocation and not automatically a JMH measurement iteration. Iterations within one fork share compilation history, heap state, host conditions, and often temporal autocorrelation. Treating them as independent inflates the sample size and confidence.

Use one of these designs:

• Paired/interleaved A/B: run baseline and candidate in randomized order on the same host block. Analyze within-block effects. This cancels much host and time drift and is the preferred confirmation design when both commits can be built safely.
• Independent controlled runs: allocate independent forks/sessions to each version and compare at that level. Use when pairing would introduce carry-over or cannot be deployed.
• Historical screen: compare against a compatible distribution from trusted trunk runs. Cheap, but vulnerable to drift and selection bias; confirm borderline or blocking results.

Randomize order, retain block/run identifiers, and separate warm-up from measurement. If thermal state, CPU quota, noisy neighbors, power policy, or clock behavior dominate the effect, more iterations inside the same contaminated run do not repair the design.

Calibrate capability, not folklore thresholds

Collect unchanged-code runs across the times, hosts, and restart boundaries the gate will experience. Then inject representative regressions near the MPIR. Estimate:

• false-block rate under no change;
• decision probabilities at the MPIR boundary and power at declared larger effects;
• inconclusive frequency and retry cost;
• host/day/JDK variance and outlier behavior;
• sensitivity to order, warm-up, thermal state, and dependency drift.

Choose repetition count and rule by simulation, bootstrap at the independent-unit level, or a model that represents the blocking structure. max/min over ten runs is an unstable range, not a reusable noise estimate. “Two times the noise” and “four times the noise” do not define a false-positive rate or power.

Environment compatibility

Capture an environment/configuration fingerprint with every result:

• benchmark artifact and source commits, dependency lock/checksum, dataset checksum;
• JMH version, benchmark mode, unit, parameters, threads, forks, warm-up and measurement;
• complete Java runtime identity and effective JVM arguments;
• OS/kernel, CPU model/topology, cgroup/quota/cpuset, memory limit, host/runner pool;
• collector, heap policy, power/frequency policy, and relevant services or agents.

Pin values only when the experiment intends to hold them constant. Fixed heap sizes, AlwaysPreTouch, a collector, or CPU affinity can improve repeatability, but can also make the gate unlike production or remove the phenomenon under test. Maintain separate “controlled microbenchmark” and “target-like system” profiles when both questions matter.

Compatibility is a policy, not exact string equality for every field. Classify changes as:

• comparable;
• comparable after a proven conversion or model adjustment;
• new calibration epoch;
• invalid comparison.

A JDK, hardware, collector, dependency, harness, or dataset change usually starts a new epoch. Do not label that discontinuity a code regression.

Multi-benchmark decisions

Parameter points should cover decision-relevant operating regions, not mechanically “small, medium, large” for every benchmark. Scaling claims require enough sizes and a model; one changed point alone does not prove a complexity-class change.

Testing many benchmarks, parameters, and metrics increases the chance of a false alert. Choose one explicitly:

• a small predeclared critical set with family-wise control;
• false-discovery-rate control for a broader diagnostic suite;
• a hierarchical/global screen followed by corrected drill-down;
• per-benchmark guardrails whose joint false-block behavior was calibrated empirically.

Never hide multiplicity by reporting only the worst delta. Preserve all results and disclose selection rules.

Baseline lifecycle

Treat a baseline as a versioned release artifact, not a mutable cache entry. It needs commit, fingerprint, calibration epoch, creation workflow, retention, digest, and promotion record.

Useful policies:

• Champion baseline: compare to a manually or automatically promoted trusted trunk revision; stable but can become stale.
• Recent compatible distribution: detects deviations from current trunk; handles ordinary evolution but can ratchet gradual degradation.
• Both: block against an absolute guardrail and compare with both champion and recent history; usually the most informative production policy.

Only a trusted workflow may promote. Untrusted pull-request code may read an explicitly non-secret baseline but must not overwrite it, publish a trusted result, or execute with privileged credentials. A missing, expired, malformed, or incompatible baseline yields inconclusive/invalid, never a fabricated first-run pass.

Pipeline architecture

cheap PR screen
  -> pass: report evidence
  -> clear regression: controlled confirmation
  -> inconclusive: controlled confirmation or explicit human policy

controlled confirmation
  -> pass / regression / inconclusive
  -> immutable result + provenance + diagnostics

trusted trunk workflow
  -> validate compatibility and health
  -> promote baseline according to policy

This two-stage design keeps feedback affordable without pretending a noisy executor can resolve small effects. A blocking gate needs a bounded policy for infrastructure failure and inconclusive results; silently retrying until green introduces optional-stopping bias.

Failure protocol

Use distinct machine-readable statuses and stable exit codes, for example:

Attempt to emit a report artifact before returning the code; kills/disk failures may prevent it. If output is piped through tee under Bash, temporarily disable errexit, run the pipeline, immediately capture the entire PIPESTATUS array, restore errexit, publish comparator and logging statuses, and apply the artifact-failure policy. ${PIPESTATUS[0]} alone cannot execute after a failed pipeline when the shell was launched with -e and pipefail.

Troubleshooting

Anti-patterns

Anti-pattern: threshold from a customary percentage. It happens because a single number is easy to implement. It is dangerous because neither practical importance nor error rates are known. Detect it by the absence of MPIR, calibration data, and power. Replace it with a predeclared decision plus empirical operating-characteristic study.

Anti-pattern: non-overlapping JMH scoreError intervals as a two-version test. It happens because JSON exposes scoreError. That field describes JMH's aggregate estimate (for AVG, current JMH source uses a 99.9% mean-error calculation); interval overlap is not a general paired or independent two-sample test and ignores run blocking. Keep the field for reporting, but analyze retained independent-unit observations using the declared design.

Anti-pattern: retry until green. It converts noisy evidence into selection bias and makes false passes likely. Bound retries in advance, retain every attempt, and return inconclusive when the evidence budget is exhausted.

Anti-pattern: always use dedicated hardware. Dedicated hosts can reduce variance but may still drift and may not be needed for coarse regressions. Measure resolution and cost. Use a shared screen plus controlled confirmation when that meets the decision need.

Review checklist

• Metric, direction, unit, parameters, MPIR, and absolute guardrail are explicit.
• Independent experimental unit and pairing/blocking are explicit.
• False-block rate, power, inconclusive rate, and runtime were calibrated.
• Comparator rejects incompatible fingerprints and missing entries.
• Multiplicity and retries have predeclared policies.
• Baselines are immutable, attributable, epoch-aware, and promoted only by trust.
• Pipeline preserves report artifacts and exact comparator status on every path.
• Injected regressions, no-change controls, malformed input, missing baseline, and infrastructure failure were tested end to end.
• A green result is worded as bounded evidence, not proof of equal performance.

References

• Calibrating the gate — experimental design, calibration, inference choices, and baseline drift.
• Pipeline construction — result contract, safe shell status capture, workflow trust boundaries, and end-to-end tests.
• OpenJDK JMH project — authoritative harness source and samples; pin the version used by the repository.
• JMH Result source — current definitions of score, scoreError, confidence, and sample count; verify against the pinned JMH tag.
• GitHub Actions workflow syntax — documented shell invocation and fail-fast behavior.
• GitHub Actions dependency-cache reference — cache scope and low-trust security model.