Performance Testing

Performance Testing

Concept of the skill

Performance testing is the discipline of measuring a system's non-functional properties — latency, throughput, error rate, resource utilization, saturation — by running the system under controlled load and observing the resulting metrics. Where functional tests answer "does the system produce the right output?", performance tests answer "does the system produce the right output quickly enough and at sufficient scale, while staying within resource budgets and error tolerances?" The unit of judgment is whether measured metrics meet defined acceptance thresholds — typically Service-Level Objectives (SLOs) expressed as percentiles (e.g., "p95 latency below 200ms at 1,000 RPS sustained for 30 minutes").

Replaces "is it fast enough?" answered by intuition or user complaints with empirical answers measured against an explicit SLO. Solves the problem that performance is multi-dimensional — latency distribution, throughput ceiling, error rate under load, resource utilization, saturation point are each separate properties; each requires its own measurement; each can be acceptable while others fail. Reducing performance to a single number (especially average latency) is the most common discipline failure. Sub-purpose: verify that declared performance budgets (per performance-budgets) hold under realistic load — without performance tests, a budget is an aspirational threshold without empirical evidence; without budgets, a performance test produces measurements without a verdict. A complete pre-launch performance test suite runs all six load shapes because each verifies a different property; an ongoing suite typically runs smoke on every PR, load on every merge, with stress/spike/soak/breakpoint on cadence.

Distinct from performance-engineering, which owns the activity of profiling and optimizing a specific slow path once it has been identified — this skill owns the discipline of exercising the system under controlled load to discover and quantify performance behavior; the two compose (this skill produces measurements that locate bottlenecks and verifies optimizations afterward). Distinct from performance-budgets, which owns the declaration of the threshold-and-consequence contract (metric, threshold, percentile, consequence) as a quality property — this skill owns the test mechanism that exercises the system under load and verifies the budgets hold. Distinct from observability, which owns real-time runtime measurement of the deployed system — this skill is offline controlled measurement; the two compose (one for pre-deploy verification, the other for production runtime validation). Distinct from chaos-engineering (fault injection in deployed systems), microbenchmarks (single-function isolation via language benchmark tools), testing-strategy (level/scope), integration-test-design and e2e-test-design (correctness across seams or user journeys — this skill exercises those same paths under load), and mutation-testing (test-suite quality measurement). Performance testing is to a software system what a load-bearing inspection is to a bridge — you do not certify a bridge by walking across it (functional test) and concluding it works; you drive trucks of known weight across at increasing volumes, with strain gauges on every beam, and verify the deflection stays within spec under expected traffic, that the failure mode is graceful when overloaded (cracks before collapse), that nothing creeps over a long soak. A bridge whose 'average' load it can carry is 50 tonnes but whose p99 stressor reveals harmonic resonance at 80 tonnes is the bridge that fails on a windy day. The wrong mental model is that average latency is a meaningful performance metric. It is not. A system whose mean is 50ms and p99 is 5 seconds has a user-experience problem the mean hides — 1 in 100 users sees a 5-second response time. Acceptance criteria should always be percentiles (or distributions); averages are easily skewed by outliers in both directions. Adjacent misconceptions: that performance tests in a stripped/non-production environment are meaningful (they are not — the test produces a measurement of that environment, not the system; production hardware, network, data volumes, dependency versions, and configuration are the load-bearing investment); that one-time pre-launch testing is enough (it is not — regressions accumulate; the ongoing value is continuous testing in CI); that load tests alone characterize the system (they do not — a complete pre-launch suite runs all six shapes, because each verifies a different property: load verifies SLO at design load, stress verifies graceful failure, spike verifies elasticity, soak verifies no leaks or memory growth, breakpoint quantifies the capacity ceiling); that performance testing replaces observability (it does not — performance testing is offline controlled measurement; observability is online real-traffic measurement; the two compose); that coordinated omission in the load tool doesn't matter (it does — Tene's canonical talk shows naive percentile reporting silently drops slow responses; honest tools account for it); and that a performance test without an SLO is a verification (it is not — without an SLO it is a measurement without a verdict; the SLO is what makes the test a verification rather than an information report).

Coverage

The discipline of measuring non-functional system properties — latency, throughput, error rate, resource utilization, saturation — by running the system under controlled load and verifying against explicit SLO thresholds. Covers the five primitives (load profile, workload, latency metric, throughput metric, SLO target), the six load-shape types (smoke, load, stress, spike, soak, breakpoint) that each test a different property, the latency-percentile discipline (p50/p95/p99/p99.9) that replaces the misleading average, the environment-fidelity requirement that makes results meaningful, the modern tool landscape (k6, JMeter, Locust, Gatling, Vegeta, Wrk), and the distinction from observability (offline controlled vs online real-traffic) and benchmarking (system vs single-function).

Philosophy of the skill

Performance testing is the discipline of adding the time and scale dimensions to verification. Functional tests verify "the system produces the right output"; performance tests verify "the system produces the right output quickly enough and at sufficient scale, within resource budgets and error tolerances." Without performance testing, "is it fast enough" is answered by intuition or by users complaining; with performance testing, it is answered by measurement against an explicit SLO.

The central insight is that performance is multi-dimensional. Latency distribution, throughput ceiling, error rate under load, resource utilization, saturation point — each is a separate property; each requires its own measurement; each can be acceptable while others fail. Reducing performance to a single number (especially average latency) is the most common discipline failure.

The complementary insight is environment fidelity. A performance test in a non-production-like environment produces a measurement of that environment, not the system. Production hardware, network, data volumes, dependency versions, and configuration are the load-bearing investment for performance testing; the load tool itself is increasingly cheap.

The Six Load Shapes

A complete pre-launch performance test suite runs all six. An ongoing test suite usually runs smoke on every PR and load on every merge, with stress / spike / soak / breakpoint on cadence.

Latency Percentiles — The Honest Vocabulary

Acceptance criteria should always be percentiles (or distributions), never averages. A system whose mean is 50ms and p99 is 5 seconds has a user-experience problem the mean hides.

SLO-Driven Performance Tests

A performance test without an SLO is "we measured X" without a verdict. An SLO-driven performance test is:

SLO statement:
  - p95 latency < 200ms
  - p99 latency < 500ms
  - error rate < 0.1%
  - sustained throughput >= 1000 RPS

Test design:
  - Load shape: constant 1000 RPS for 30 minutes
  - Workload: 70% reads, 25% writes, 5% complex queries
  - Environment: production-equivalent staging
  - Pass: all four SLO conditions met for full test duration
  - Fail: any SLO condition violated for > 60 seconds cumulative

The SLO is what makes the test a verification rather than a measurement.

Tool Selection

Verification

After applying this skill, verify:

• Performance tests have explicit SLOs as acceptance criteria. "We measured X" without a target is not a verification.
• Acceptance criteria are stated as percentiles (p50/p95/p99/p99.9), never as averages. Tests that report only averages are misleading.
• The test environment is production-equivalent in hardware, network, data volumes, dependency versions, and configuration. Stripped environments produce informational results, not verifications.
• A range of load shapes is tested: smoke (PR-level), load (merge-level), stress / spike / soak / breakpoint (pre-launch and on cadence).
• Workload composition is derived from production traffic patterns where possible. Synthetic load with the right RPS but wrong operation mix tests the wrong code paths.
• The load tool is not the bottleneck. Distributed load generation is used where single-generator capacity might be exceeded.
• Performance tests run continuously (not just pre-launch). Regression detection is the on-going value; one-time tests are point-in-time signals only.
• Soak tests run for hours and exercise the leak and degradation failure modes that shorter tests miss.
• Performance testing is paired with observability for production validation. The two compose; neither replaces the other.
• Stress and breakpoint tests are run to characterize failure mode and capacity ceiling. A system whose failure mode is unknown cannot be operated.

Do NOT Use When

Key Sources

• Grafana Labs / k6 Team. "k6 Documentation". Canonical reference for the modern JavaScript-scriptable load testing tool; covers thresholds, metrics, scenarios, and common load shapes.
• Grafana Labs / k6 Team. "Thresholds". Current reference for pass/fail criteria in k6 load tests.
• Grafana Labs / k6 Team. "Automated performance testing". Current reference for smoke, average/load, stress, spike, and soak automation guidance.
• Apache Software Foundation. "Apache JMeter — User Manual". Canonical reference for the most-established cross-protocol performance testing tool.
• Locust Team. "Locust Documentation". Reference for the Python-scriptable distributed load testing tool.
• Gatling Team. "Gatling Documentation". Reference for the high-performance Scala-based load testing tool.
• Dean, J., & Norvig, P. "Latency Numbers Every Programmer Should Know". Industry-canonical reference table for the latency scales that performance testing measures against.
• Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site Reliability Engineering. O'Reilly. Google SRE book; chapter on SLOs as the contract performance tests verify against.
• Brendan Gregg. "The USE Method: A method for analyzing system performance". The Utilization-Saturation-Errors framework that underlies systematic performance analysis.
• Tene, G. "How NOT to Measure Latency". The canonical talk on percentile latency, coordinated omission, and the misleading nature of average-latency reporting.
• Smith, C. U., & Williams, L. G. (2002). Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley. Foundational reference on performance engineering methodology.
• Kounev, S., Lange, K.-D., & von Kistowski, J. (2020). Systems Benchmarking: For Scientists and Engineers. Springer. Modern reference on performance testing methodology and benchmark design.