Lifecycle: draft
Performance Regression Analysis
Purpose and scope
Determine whether a change materially improves the performance property that matters without trading away correctness, safety, reliability, or maintainability unintentionally. This skill covers measurement and decision-making; it does not replace capacity planning, production incident response, or domain-specific performance modeling.
Triggers and prerequisites
Trigger when a task says faster, lower latency, higher throughput, cheaper, more efficient, less memory, or when a performance regression is suspected. Prerequisites: a defined workload, target metric and percentile, baseline revision or measurement, environment, resource constraints, correctness criteria, and an acceptable measurement method.
Decision criteria
- Performance is a property of a workload and environment, not an intrinsic label on an implementation.
- Define the metric, unit, percentile or distribution, steady-state/warm-up behavior, sample size, and practical significance before comparing results.
- A speedup is not an improvement if it changes semantics, increases error rate, worsens tail latency, exhausts resources, weakens security, or shifts cost elsewhere.
- Prefer a measured bottleneck and reversible change over optimization by convention or intuition.
Procedure
- Translate the request into a measurable target: workload, operation, metric, threshold, percentile, resource budget, and correctness constraints. Identify what must not regress.
- Establish a baseline at a known revision. Use representative inputs, realistic data size and distribution, stable environment, warm-up policy, and enough repetitions to characterize variance.
- Form hypotheses about the bottleneck from profiling, traces, resource measurements, or algorithmic analysis. Separate measured observations from guesses and identify at least one alternative explanation.
- Design the smallest experiment that distinguishes the hypotheses. Change one material variable where practical, isolate confounders, and avoid benchmarking a synthetic path that does not represent the claim.
- Implement the narrowest optimization. Preserve output, error, ordering, authorization, durability, concurrency, and resource-safety behavior; keep unrelated refactors separate.
- Measure baseline and candidate under the same protocol. Report distributions and variance, not only a favorable run or average. Check CPU, memory, I/O, network, allocations, energy, cost, and error rate when relevant.
- Validate boundary workloads: empty and large inputs, cache hit/miss, cold/warm state, concurrency, contention, timeouts, partial failure, and adversarial or worst-case inputs as applicable.
- Investigate any tradeoff or unexplained difference. Reject benchmark gaming, input-specific wins presented as general improvement, and changes whose practical significance is smaller than measurement noise.
- Record workload, environment, revisions, commands or tooling, raw/summary results, limitations, decision, rollback path, and the trigger for remeasurement. Add a durable benchmark or regression guard only when its maintenance cost is justified.
Examples and counterexamples
Good: A query optimization is evaluated on representative data with cold and warm caches, p50 and p99 latency, error rate, memory, and result equivalence before acceptance.
Bad: Report one local happy-path timing as proof that the query is faster in production.
Good: A cache reduces average latency but increases stale-data risk; the decision records the freshness boundary and rejects the change where that tradeoff is unacceptable.
Bad: Optimize a loop until a microbenchmark improves while ignoring increased memory use, tail latency, or the real workload.
Failure modes and recovery
If no representative workload or baseline exists, narrow the claim and build measurement first; do not invent a win. If results vary materially, investigate environment, warm-up, contention, randomness, and workload distribution before selecting a favorable sample. If profiling is inconclusive, preserve uncertainty and test competing hypotheses. If a performance change alters correctness or safety, treat it as a separate behavioral change and route it through the appropriate review skills.
Validation evidence and provenance
Claims in this skill map to graded findings in docs/research.md:
- SLOs and error budgets define the measurement contract (Q6, Moderate): Google SRE SLO chapter; practitioner consensus.
- Representative workloads and before/after baselines (Q6-built methodology, Principled): design rules on top of the SLO frame.
Source boundary: the sources do not evaluate benchmarking procedures for specific stacks; workload realism must be verified per repository.
Confidence: medium. Freshness: review when the research base, workload tooling, or SLO practice changes.
Disconfirmation: evidence that synthetic regression benchmarks reliably predict field performance would justify lighter procedures; evidence they diverge strengthens the representativeness requirement.
Related skills and conflicts
Related: evidence-driven-debugging, observability-and-instrumentation, behavior-preserving-refactoring, differential-patch-review, regression-test-design, secure-coding-review, and repository-change-verification. This skill does not authorize weakening correctness, security, or reliability to obtain a benchmark improvement.
1---2name: performance-regression-analysis3description: Investigate and validate software performance changes using representative workloads, baselines, variance-aware measurements, and resource tradeoffs. Use when an AI coding agent is asked to make code faster, reduce latency or cost, increase throughput, or diagnose a performance regression.4---56Lifecycle: `draft`78# Performance Regression Analysis910## Purpose and scope1112Determine whether a change materially improves the performance property that matters without trading away correctness, safety, reliability, or maintainability unintentionally. This skill covers measurement and decision-making; it does not replace capacity planning, production incident response, or domain-specific performance modeling.1314## Triggers and prerequisites1516Trigger when a task says faster, lower latency, higher throughput, cheaper, more efficient, less memory, or when a performance regression is suspected. Prerequisites: a defined workload, target metric and percentile, baseline revision or measurement, environment, resource constraints, correctness criteria, and an acceptable measurement method.1718## Decision criteria1920- Performance is a property of a workload and environment, not an intrinsic label on an implementation.21- Define the metric, unit, percentile or distribution, steady-state/warm-up behavior, sample size, and practical significance before comparing results.22- A speedup is not an improvement if it changes semantics, increases error rate, worsens tail latency, exhausts resources, weakens security, or shifts cost elsewhere.23- Prefer a measured bottleneck and reversible change over optimization by convention or intuition.2425## Procedure26271. Translate the request into a measurable target: workload, operation, metric, threshold, percentile, resource budget, and correctness constraints. Identify what must not regress.282. Establish a baseline at a known revision. Use representative inputs, realistic data size and distribution, stable environment, warm-up policy, and enough repetitions to characterize variance.293. Form hypotheses about the bottleneck from profiling, traces, resource measurements, or algorithmic analysis. Separate measured observations from guesses and identify at least one alternative explanation.304. Design the smallest experiment that distinguishes the hypotheses. Change one material variable where practical, isolate confounders, and avoid benchmarking a synthetic path that does not represent the claim.315. Implement the narrowest optimization. Preserve output, error, ordering, authorization, durability, concurrency, and resource-safety behavior; keep unrelated refactors separate.326. Measure baseline and candidate under the same protocol. Report distributions and variance, not only a favorable run or average. Check CPU, memory, I/O, network, allocations, energy, cost, and error rate when relevant.337. Validate boundary workloads: empty and large inputs, cache hit/miss, cold/warm state, concurrency, contention, timeouts, partial failure, and adversarial or worst-case inputs as applicable.348. Investigate any tradeoff or unexplained difference. Reject benchmark gaming, input-specific wins presented as general improvement, and changes whose practical significance is smaller than measurement noise.359. Record workload, environment, revisions, commands or tooling, raw/summary results, limitations, decision, rollback path, and the trigger for remeasurement. Add a durable benchmark or regression guard only when its maintenance cost is justified.3637## Examples and counterexamples3839Good: A query optimization is evaluated on representative data with cold and warm caches, p50 and p99 latency, error rate, memory, and result equivalence before acceptance.4041Bad: Report one local happy-path timing as proof that the query is faster in production.4243Good: A cache reduces average latency but increases stale-data risk; the decision records the freshness boundary and rejects the change where that tradeoff is unacceptable.4445Bad: Optimize a loop until a microbenchmark improves while ignoring increased memory use, tail latency, or the real workload.4647## Failure modes and recovery4849If no representative workload or baseline exists, narrow the claim and build measurement first; do not invent a win. If results vary materially, investigate environment, warm-up, contention, randomness, and workload distribution before selecting a favorable sample. If profiling is inconclusive, preserve uncertainty and test competing hypotheses. If a performance change alters correctness or safety, treat it as a separate behavioral change and route it through the appropriate review skills.5051## Validation evidence and provenance52Claims in this skill map to graded findings in [`docs/research.md`](../../docs/research.md):5354- SLOs and error budgets define the measurement contract (Q6, Moderate): Google SRE SLO chapter; practitioner consensus.55- Representative workloads and before/after baselines (Q6-built methodology, Principled): design rules on top of the SLO frame.5657Source boundary: the sources do not evaluate benchmarking procedures for specific stacks; workload realism must be verified per repository.5859Confidence: medium. Freshness: review when the research base, workload tooling, or SLO practice changes.6061Disconfirmation: evidence that synthetic regression benchmarks reliably predict field performance would justify lighter procedures; evidence they diverge strengthens the representativeness requirement.62## Related skills and conflicts6364Related: `evidence-driven-debugging`, `observability-and-instrumentation`, `behavior-preserving-refactoring`, `differential-patch-review`, `regression-test-design`, `secure-coding-review`, and `repository-change-verification`. This skill does not authorize weakening correctness, security, or reliability to obtain a benchmark improvement.
ChloeVPin