Performance engineering standards
Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).
1. Scope and triggers
Applies to server-side performance methodology: system, runtime, service and the path that
joins them. What is measured, with which statistic, with which load generator, how it is
profiled, in what order the culprits are hunted and what evidence is needed to accept that an
optimisation is an improvement and not a coincidence. The thesis: measure before touching,
and know what to measure. An optimisation without measurement before and after is not
engineering, it is superstition with a commit.
Triggers: perf record/perf report/perf script/perf stat, perf_event_paranoid,
flamegraph.pl, stackcollapse-perf.pl, flame graph, off-CPU, bpftrace, eBPF, BCC,
pprof, net/http/pprof, go tool pprof, py-spy, async-profiler, JFR, jemalloc /
heaptrack / massif, Pyroscope, Parca, Profiles Drilldown, continuous profiling, OTLP
Profiles, wrk/wrk2 -R, vegeta -rate, k6 constant-arrival-rate / ramping-arrival-rate
/ VU, Gatling injectOpen / injectClosed / rampUsersPerSec, JMeter Open Model Thread
Group, Locust, ab, HdrHistogram, coordinated omission, tail latency, p95/p99/p99.9,
USE, RED, golden signals, Little's law, Amdahl's law, latency budget, load / stress / soak
/ spike testing, N+1, cold start, GC pause.
Not applicable: see web-performance-standards (the finest boundary in the catalogue,
and it is already written from the other side: theirs are the browser and the real user
—Core Web Vitals, RUM at p75, bundle budget, hydration cost, CLS—; here the server, the
runtime and the system. The contact point is TTFB: it is their input metric and my
output metric — how server response time is reduced belongs here; that this TTFB feeds into
LCP and how much it weighs in the field p75 is theirs. Operational corollary: an impeccable
server p99 does not contradict a slow website, and vice versa; a debate about one is never
closed with the other's number), sre-practice-standards (important boundary: SLO, SLI,
error budget, burn rate alerts, capacity planning and production reliability are theirs.
Rule: "how much latency can we afford and what do we do if we exceed it?" belongs to
sre-practice; "why is it slow and what fixes it?" belongs here. The SLO defines the
objective; this skill provides the method that meets it), observability-standards (the
telemetry platform is theirs: collectors, OTel Collector, storage, retention, trace sampling,
cardinality, dashboards and alerts — continuous profiling is ingested and stored there;
here what is profiled, at what sampling frequency and how a flame graph is read),
testing-qa-standards (reciprocal declared in both directions: there the load test is
planned as a test type —scenarios, data, stopping criteria, and the rule that it never goes
in the PR gate—; here the load model (open or closed), the measurement methodology and the
interpretation of the result, §4. A load script written there that uses a closed model
against an open system produces an invalid number according to this skill),
data-platform-standards, sql-standards, mysql-mariadb-dba-standards,
oracle-dba-standards, sqlserver-dba-standards, nosql-standards, timeseries-db-standards
and search-engines-standards (engine, index and query plan tuning are theirs; here only
the methodology that proves the bottleneck is in the database and with what evidence you knock
on their door), python-standards, go-standards, rust-standards,
jvm-spring-standards, dotnet-standards, cpp-standards, c-standards,
typescript-standards and the other language skills (the runtime's specific profiler, its
flags and its garbage collector tuning are theirs; here the common method and when to reach
for them), gpu-computing-standards (kernels, occupancy and GPU memory),
caching-cdn-standards (HTTP cache policy, the CDN and purging; here the cache as an
architecture decision with a coherence cost, §6.5), networking-standards and
network-troubleshooting-standards (network diagnosis is theirs: RTT, loss, MTU,
retransmits, capture; here only placing the network in the suspect tree and handing them the
case), kubernetes-standards (requests/limits, HPA, cgroup CPU throttling),
linux-administration-standards and linux-storage-standards (system and storage tuning),
operating-systems-standards ("how is it measured?" belongs here; "what OS mechanism
produces that number?" is theirs — scheduler, memory management, the real cost of a system
call, fsync guarantees), finops-standards (cost as a metric and its budget; here cost
only appears as an argument for sizing) and microservices-architecture-standards (the
division of responsibilities between services; here the division of the latency budget
between them, §3.3), refactoring-tech-debt-standards (optimising is not refactoring:
refactoring changes the structure without changing observable behaviour; optimising changes
an observable characteristic —latency— and therefore is measured before and after and is
justified with that data. Debt criteria, its register and its funding are theirs; the
methodology of finding and proving the improvement, here. And the warning they share:
optimised code expires and needs a review date, whereas a well-done refactor does not),
green-it-standards (the methodology of measuring before touching,
profiling and proving the improvement belongs here; the energy and carbon metric is
theirs, with its own accounting and its own traps. Warning both share: code efficiency and
footprint reduction are not the same thing — the real order of impact starts with switching
off what is idle and right-sizing, and optimising code for sustainability is only justified at
a scale that has to be calculated, not inherited).
2. Default decisions
Verify the latest version and the licence on the web before pinning it in a real project (§8).
2.1 The two named methods, cited at source
They are not styles: they are checklists with an author, and they apply to different
things. Using them crosswise is the most expensive framing error.
USE — Brendan Gregg (brendangregg.com/usemethod.html; published on his dtrace.org blog
on 29 Feb 2012 and in ACM Queue as Thinking Methodically about Performance, 2012, and in
CACM, 2013). Verbatim:
"For every resource, check utilization, saturation, and errors."
"resource: all physical server functional components (CPUs, disks, busses, ...)"
"utilization: the average time that the resource was busy servicing work"
"saturation: the degree to which the resource has extra work which it can't service, often queued"
"errors: the count of error events"
Scope declared by the author himself, verbatim: "It solves about 80% of server issues with 5%
of the effort", and its limit, also verbatim: "There are many problem types it doesn't
solve, which will require other methods and longer time spans" / "While the USE Method may
find 80% of server issues, latency-based methodologies (eg, Method R) can approach finding 100%
of all issues." The 80/5 is cited as what it is —a claim by the author, not a study— or it
is not cited at all.
RED — Tom Wilkie, created in 2015 (Grafana Labs, The RED Method: How to Instrument Your
Services, presented at GrafanaCon EU 2018; Wilkie arrived at Grafana with the acquisition of
Kausal, and developed RED at Weaveworks). Verbatim from the definition:
"Rate (the number of requests per second), Errors (the number of those requests that are
failing), Duration (the amount of time those requests take)"
Declared reason, verbatim: "The USE Method doesn't really apply to services; it applies to
hardware, network disks, things like this. We really wanted a microservices-oriented monitoring
philosophy, so we came up with the RED Method." And the relationship with the two neighbours,
verbatim: on the Four Golden Signals (latency, traffic, errors, saturation), "This is
basically the same as the RED Method, but includes saturation"; and on using them together,
"the RED Method is about caring about your users and how happy they are... and the USE Method
is about caring about your machines and how happy they are... They're complimentary."
⚠ Discrepancy declared in the source itself: the Grafana article heads both lists with
"For every resource, monitor:", including RED's, despite the text immediately before it
arguing that RED exists precisely because USE does not apply to services. The formulation
consistent with the rest of the article and with universal usage is "for every service".
When citing RED, cite the triad, not the heading.
Rule of use: USE for resources (CPU, memory, disk, network, controllers, cgroups) and
RED for services and endpoints. In an incident you walk both: RED says which service
hurts, USE says which resource causes it. The SRE Book's golden signals are RED +
saturation and belong to sre-practice-standards; here they are used as a search index, not
as an alert.
2.2 Tools — status and licence verified as of Aug 2026
| Use |
Default |
Licence / verified status |
Note |
| CPU profiling on Linux |
perf |
GPL-2.0 (SPDX-License-Identifier: GPL-2.0 in tools/perf of the kernel tree) |
Sampling, in the kernel; no third-party dependencies |
| Profile visualisation |
FlameGraph (stackcollapse-*.pl + flamegraph.pl) |
⚠ CDDL — "all files in this distribution are released under the Common Development and Distribution License (CDDL)" (docs/cddl1.txt) |
It is not MIT nor Apache: per-file copyleft. If it gets packaged, it goes through review |
| Ad hoc dynamic tracing |
bpftrace |
Apache-2.0 (raw LICENSE) — v0.26.1 (2 Jun 2026) |
For questions no counter answers |
| Continuous profiling (OSS) |
Grafana Pyroscope |
⚠ AGPL-3.0 (raw LICENSE) — active weekly releases (Aug 2026) |
AGPL: relevant if offered as a service. UI: Profiles Drilldown (plugin, public preview, not GA) |
| Continuous profiling (alternative) |
Parca |
Apache-2.0 (raw LICENSE) — active, engine behind Polar Signals Cloud |
Permissive licence; smaller ecosystem |
| Interchange standard |
OTLP Profiles |
⚠ Public alpha since 26 Mar 2026 |
§2.4. Do not bet critical production on it yet |
| Constant-rate load |
wrk2 (-R mandatory) |
Apache-2.0 (raw LICENSE) |
Historical reference for corrected latency; old repo, verify before adopting |
| Programmable load |
k6 with constant-arrival-rate / ramping-arrival-rate executor |
⚠ AGPL-3.0 (raw LICENSE.md) — v2.1.0 (30 Jun 2026) |
The VU default is a closed model: the executor has to be chosen by hand (§4.2) |
| JVM load |
Gatling (injectOpen) |
Apache-2.0 (LICENSE.txt; "Gatling Open Source is licensed under Apache 2.0") — v3.15.1 (25 May 2026) |
Gatling Enterprise is a separate paid product |
| Legacy GUI load |
JMeter |
Apache-2.0 — 5.6.3, no new release since Jan 2024 |
Open Model Thread Group declared experimental (§4.2). It is not the default for anything new |
| Simple HTTP attack |
vegeta (-rate) |
MIT (raw LICENSE) |
Good for one endpoint, not for a user journey |
| Load in Python |
Locust |
MIT (raw LICENSE) |
Requires explicitly choosing the arrival model |
| Latency distribution |
HdrHistogram |
— |
Latency recording is always a histogram, never a running average (§3.2) |
Warnings that correct common assumptions:
- k6 is AGPL-3.0, not Apache or MIT. If it is embedded in a product or offered as a
service, that is a legal decision, not a tool choice.
- Pyroscope is AGPL-3.0; Parca is Apache-2.0. They are the same category with licences
incompatible in purpose: the choice is made with the lawyer present, not by the UI.
- FlameGraph is CDDL, not permissive. It is the case most people assume is MIT.
- JMeter has gone more than two years without a release on its official site (5.6.3, Jan
2024). The GitHub feed agrees here, but the GitHub feed is not the source of truth: Apache
publishes at
jmeter.apache.org. Check there (§8).
2.3 Sampling versus instrumentation
|
Sampling |
Instrumentation (tracing/instrumenting) |
| What it does |
Interrupts at a fixed frequency and records the stack |
Records every event that has been marked |
| Bias |
Statistical: misses the rare, sees the expensive well |
Observer bias: the cost grows with the event's frequency |
| Cost |
Bounded and predictable by frequency |
Unbounded: an instrumented hot function can dominate its own profile |
| Use |
By default, and the only acceptable one in continuous production |
Ad hoc, bounded, on the already suspect path |
| Trap |
Does not see off-CPU time (I/O wait, blocking, scheduler) |
A counter per call turns the cheap into the expensive and changes the result it was going to measure |
Hard rules:
- In production you profile by sampling. Fine-grained instrumentation in production only
with bounded scope, a time window and automatic deactivation.
- Frequency by convention: non-round values (e.g.
perf record -F 99 instead of
100 Hz). A frequency that coincides with a timer, a tick or a system loop always samples
the same phase and produces a biased profile that looks perfect.
- A CPU profile does not explain a waiting latency. If the service is slow but the CPU is
idle, the CPU profile will come out empty of culprits: you need an off-CPU profile
(blocking, scheduler, I/O) or traces. It is the most frequent diagnostic error after
coordinated omission.
- A flame graph is read by width, not by height. Height is stack depth; width is time. A
tall, narrow stack costs nothing.
- Profiling CPU and profiling memory allocation are two different questions. The
allocation/heap profile is the one that explains GC pressure and many tail latencies; the
CPU one does not see it. Both, or the diagnosis is incomplete.
2.4 OpenTelemetry Profiles — verified status, and why it matters
Status: public alpha since 26 Mar 2026 (official OpenTelemetry announcement, OpenTelemetry
Profiles Enters Public Alpha, signed by the Profiling SIG). What is verifiable from it:
- Format with stack deduplication and dictionary tables; optional link to
trace_id /
span_id for cross-signal correlation (profile ↔ trace); on-wire size reduction cited by
the announcement itself as "40% smaller wire size" for the string dictionary.
- Relationship with
pprof, verbatim: "Originally inspired by the pprof format and developed in
collaboration with pprof maintainers, OTLP Profiles has evolved into an independent standard...
Data in the original pprof format can be round-trip converted to/from OTLP Profiles with no
loss of information."
- The whole-system eBPF profiler donated by Elastic is already an official Collector
component (receiver), with on-target symbolisation for Go, ARM64 for Node.js/V8, initial
BEAM and .NET 9/10 support.
- Warning from the announcement itself, verbatim: "As the signal is still under development,
production-ready backends have not yet emerged but multiple vendors are working on supporting
OpenTelemetry Profiles."
Criteria: alpha means alpha. OTLP Profiles is adopted as the target format when
designing today (it avoids getting tied to a proprietary agent), but the production backend
is still Pyroscope or Parca, and the architectural commitment is reviewed at each release
(§8). Pinning a critical backend today on the alpha signal is debt with a due date.
3. The prior discipline: define the objective before measuring
3.1 A performance objective without these four pieces is not an objective
Metric + percentile + load + hardware. Miss one and the number is not comparable with
anything, not even with itself the following week.
p99 latency of /api/orders ≤ 300 ms
at 500 req/s sustained (open arrival model, Poisson)
on 3 replicas of 2 vCPU / 4 GiB, database with the production dataset
measured at the client, not at the server
- "Fast" is not an objective. Neither is "under 300 ms", if it does not say at which
percentile nor under how much load: any system meets any latency with one user.
- The objective is expressed in user terms, not component terms: "search responds in X",
not "the query takes Y". The per-component budget is derived from the objective (§3.3),
never the other way round.
- It is measured at the client. The time the server thinks it took excludes accept
queueing, TLS, and precisely the queue that causes the problem.
3.2 Why the mean lies, and what is used instead
- The latency distribution is not normal: it has a long right tail and is usually
multimodal (cache hit / cache miss; fast path / slow path; with GC / without GC).
On a multimodal distribution the mean can fall in a valley where there is no real
request at all. A mean of 120 ms is compatible with "all of them take 120 ms" and with "90 %
take 20 ms and 10 % take a second": they are two different systems and only one is acceptable.
- The mean is insensitive to the tail by construction; the tail is exactly what the user
remembers and what saturates the system upstream.
- The median lies just the same, more discreetly. Decisions are made with p99 as a
minimum, and with p99.9 in services with internal fan-out (§3.4).
- Percentiles are not averaged and are not summed. The mean of the p99s of ten instances is
not the p99 of the set; summing the p99 of three dependencies does not give the p99 of the
total. You aggregate with histograms (HdrHistogram,
histogram_quantile over buckets),
never with arithmetic over already-computed quantiles. A dashboard that averages percentiles
is broken however pretty it looks.
- Standard deviation over a long-tailed distribution means nothing. It is not reported.
- Maximum and p100: they are recorded (they expose the worst case and pathological
outliers), not used as an objective — a single event sets the number.
3.3 Latency budget and its division
The user objective is divided among the stages of the path, and the sum of the budgets is
less than the objective, not equal to it: the margin pays for retries, variance and what does
not exist yet.
- The budget is written per dependency, with a named owner, in the repository.
- A service cannot promise a p99 better than the worst p99 on its critical path. Before
committing to a number, the floor of the synchronous dependencies is summed.
- Every added synchronous hop consumes budget permanently. Adding a network call to a hot
path is an architecture decision with a price; the price is written in the ADR
(
microservices-architecture-standards).
- Retries multiply load at the worst possible moment. A retry without backoff with
jitter and without a retry budget turns a degradation into a collapse. The pattern belongs
to reliability (
sre-practice-standards); the effect on tail latency belongs here: the
retry is the user's time.
- Timeouts are derived from the budget, they are not copied. A client timeout larger than
the stage's budget is a decorative timeout.
3.4 Tail amplification: why the internal p99 is the user's p50
If a user request opens N internal calls in parallel and waits for all of them, the
probability that none falls into the slow tail decays with N. With N=100 independent calls
and an internal p99, the probability that all 100 are below the p99 is
0.99¹⁰⁰ ≈ 0.366: that is, ~63 % of user requests touch at least one call above the
internal p99. It is arithmetic, not a study.
Operational consequences, and they are harsh:
- In fan-out services you optimise p99.9, not p99. The internal p99 is already the typical
user case.
- Reducing fan-out is a first-order latency optimisation, often bigger than any
micro-optimisation of code.
- A single slow component on the path dominates the result however fast the rest are. See
§5.1 (Amdahl).
4. Measuring: load tests that produce a valid number
4.1 Coordinated omission — the most expensive and least known measurement error
What it is. Term coined by Gil Tene. The load generator inadvertently coordinates
with the measured system: if the next send depends on the previous one having finished, when
the system stalls the generator stops sending for exactly the bad interval. Result: instead
of the N slow requests a real user would have suffered, the histogram records a single slow
sample. High percentiles come out orders of magnitude better than they were.
Why it is so toxic: it is not a failure of the measured system, it is a failure of the one
measuring, and it gets worse exactly when the system gets worse — that is, the tool lies to
you more the more the truth matters. A closed generator can report an excellent p99 for a
service that stops for whole seconds.
Canonical illustration (mechanics, not an empirical figure): target 10 req/s, normal
service of 50 ms; the system stalls for 5 s. A closed generator records one request of
~5 s and none of the ~50 it should have sent in that gap. "Corrected" latency also counts the
time each pending request should have been waiting since its expected arrival instant.
What avoids it, verified:
| Tool |
Model |
Verified status |
wrk2 |
Mandatory constant rate (-R), HdrHistogram, reports corrected and uncorrected latency |
Designed by Tene specifically against CO |
| k6 |
constant-arrival-rate / ramping-arrival-rate = open; VU in a loop = closed |
Its own documentation names it, verbatim: "In some testing literature, this problem is known as coordinated omission" — describing its closed model |
| Gatling |
injectOpen (open, rampUsersPerSec) vs. injectClosed (closed); they cannot be mixed in one scenario |
Explicit choice in the script |
vegeta |
-rate = open by design |
One endpoint, not a user journey |
| Locust |
Requires a choice; its documentation warns that a test which does not reach the throughput target shows artificially low response times |
Explicit config |
| JMeter |
Open Model Thread Group, verbatim from its documentation: "This thread group is experimental, and it might change in the future releases." |
The classic Thread Groups are closed |
ab, wrk (original) |
Closed, no correction |
Banned for measuring percentiles (§7) |
Hard rules:
- The arrival model is declared in writing in the test report. A result without a declared
open or closed model is a number without units.
- Open system → open generator. A public API, a website, a database with independent
clients are open systems: real users do not wait for the ones next to them to finish
before clicking. Measuring them with a closed generator is measuring a different system.
- The closed model is legitimate when the real system is closed: a job queue with N fixed
workers, a batch, an internal client with bounded concurrency. The rule is not "always open",
it is "the generator's model must be the system's model".
- If the generator hits its own limits (CPU, sockets, descriptors, a single node against a
cluster), the test is invalid and is discarded — it is not interpreted. The generator is
monitored with the same seriousness as the target.
- When using
wrk2, the corrected latency is reported. Publishing the uncorrected one
without saying so is publishing the error.
4.2 The four test types, and what each one answers
They are planned in testing-qa-standards; the interpretation belongs here.
| Type |
Question |
Duration |
Failure signal |
| Load |
Does it meet the objective under the expected load? |
Enough to stabilise |
The latency budget is exceeded |
| Stress |
Where is the breaking point and how does it break? |
Ramp until degradation |
It breaks in an uncontrolled way (no shed, no degradation, with cascade) |
| Endurance (soak) |
Does it degrade over time? |
Hours or days |
Growing drift in memory, descriptors, connections, latency; fragmentation; poisoned cache |
| Spike |
Does it survive an abrupt jump and recover? |
Minutes |
It does not return to the previous level after the spike — the most expensive failure and the least tested |
The soak is the one most people skip and the one that prevents the most incidents: memory
and descriptor leaks do not show up in a ten-minute test, by definition.
4.3 Validity requirements for a test
Miss one and the result is not published:
- Declared environment parity. A load test against an environment that is not
production parity does not produce a number: it produces a useless number. The factors
that break it and are not negotiable: data volume and distribution (an index over a
thousand rows always fits in memory and is always fast; the query plan can be another one
at real volume), topology (one replica versus three, with or without a real load balancer
and real proxies), machine class and noisy neighbours, real network latency between
components, cgroup limits / CPU throttling, and identical runtime configuration
(GC, pools, timeouts). If there is no parity, the result is only good for comparing
with itself across identical runs, and it is written that way in the report.
- Warm-up and discard. JIT, caches, connection pools, page tables and filesystem caches
need a warm-up period that is discarded from the result. Including start-up in the
histogram contaminates the tail with an artifact.
- Representative data with realistic cardinality. A single user, a single key or a single
product turns the test into a cache test. The access distribution matters as much as the
volume (Zipf, not uniform).
- Duration long enough to reach steady state and for the periodic events to occur: major
GC, connection rotation, cache expiry, scheduled tasks, compaction.
- Repeatability: N≥3 runs. If the variance between runs exceeds the difference you want to
demonstrate, nothing has been demonstrated.
- System under test observed during the test: USE on every resource, RED on every
service. A test that only produces the final number does not allow diagnosing anything.
- Reproducible record: artifact version, commit, configuration, dataset, tool, script,
arrival model and hardware. A result nobody can reproduce is an anecdote.
4.4 Microbenchmarks
- A microbenchmark measures the microbenchmark. It is valid for comparing two
implementations of the same function, and not for predicting the service's latency.
- The language's harness is used (it is theirs: JMH,
go test -bench, criterion,
pytest-benchmark), with warm-up and statistical significance. Without reported variance,
it is not accepted.
- The compiler can eliminate the measured code. If the result is not consumed, the
optimiser deletes it and you measure an empty loop. It is a silent failure.
- Justifying an architecture change with a microbenchmark is forbidden (§7).
5. The right order of the search
5.1 Amdahl's law: the ceiling is set by what you do not optimise
Amdahl (1967, AFIPS Conference Proceedings vol. 30, pp. 483-485 — verify the reference
before citing it in a formal document, §8). Formulation: if a fraction p of the time is
sped up by a factor s, the total improvement is 1 / ((1−p) + p/s).
Consequence that decides the order of work: making infinitely fast a part that takes 5 % of
the time produces, at most, a 5.3 % improvement. Therefore:
- Measure the whole hot path before touching anything. The distribution of time by stage
is the first artifact, always.
- Optimise in order of time fraction, not in order of ease or of interest.
- Recalculate after each change: once the bottleneck is removed, the bottleneck is another
one and the priority list changes entirely. A performance task list expires as soon as the
first item is completed.
- Scaling out does not fix what is serialised. That is the practical corollary: if 20 % of
the work is serial (a global lock, a hot table, a single service), doubling replicas does not
give 2×. Gunther's Universal Scalability Law adds the coherence term (the cost of the
replicas agreeing with each other), which can make adding capacity make performance
worse. Verify the formulation before citing it (§8).
5.2 The suspect tree, by layer and in this order
You walk it top-down, and each layer is ruled out with evidence, not with intuition:
- Unnecessary work — the optimisation that always wins. Is something being computed that
nobody looks at? Is more being requested than is used? Is a whole object being serialised to
read one field? It is removed; it is not optimised.
- Application: algorithmic complexity over the real input size; N+1 of queries or
of remote calls (the classic and the most expensive); serialisation and deserialisation
(often the biggest CPU consumer in a "network" service); data copies; locks and contention;
synchronous work that should have been asynchronous.
- Runtime: garbage collector pauses and pressure, heap size and policy, badly sized
pools (connections, threads), cold start, JIT. The specific tuning belongs to the
language skill; here only the evidence that it is there.
- Database: query without an index, changed plan, locks, pool exhaustion, long
transactions,
N+1. You knock on the engine skill's door with evidence: the query, its
plan and its share of total time.
- Network: RTT, hops, MTU, loss and retransmission, TLS handshake, DNS.
network-troubleshooting-standards diagnoses; here you only locate it and hand it over.
- Disk / storage: service latency, queue depth, random vs. sequential pattern,
fsync,
IOPS saturation.
- Kernel and platform: scheduler, cgroup throttling (the favourite cause of bad p99 on
Kubernetes with aggressive CPU limits), NUMA, descriptor exhaustion, conntrack table,
memory pressure.
Cross-cutting rule: a layer is ruled out with a measurement, not with an argument. "It
can't be the database" does not rule out the database.
5.3 Latency versus throughput: improving one worsens the other
They are not the same quantity and they are optimised with opposing techniques:
| Technique |
Effect on throughput |
Effect on latency |
| Batching |
↑↑ |
↑ (waits to fill the batch) |
| Deeper queue |
↑ (absorbs spikes) |
↑↑ (more waiting time) |
| More concurrency |
↑ until saturation |
↑ past saturation |
| Compression |
↑ if the network is the bottleneck |
↑ CPU at both ends |
| Speculative execution / hedged requests |
↓ (duplicated work) |
↓↓ in the tail |
Hard rule: which of the two is being optimised is declared before starting. One team
optimising throughput and another optimising latency on the same system undo each other's work.
And a bigger queue never fixes a capacity problem: it only turns fast errors into long
waits, which is worse.
6. Queues, saturation and sizing
6.1 Little's law
J. D. C. Little (1961), A Proof for the Queuing Formula: L = λW, Operations Research
9(3):383-387. L = mean number of units in the system, λ = mean arrival rate, W = mean time in
the system. It is independent of the distribution of arrivals, of service and of the queue
discipline: that is why it can be applied almost always.
Useful form in systems: concurrency = throughput × latency. Three calculations come out of
that, and they are done before touching configuration:
- Sizing a pool: to serve 500 req/s with 40 ms of latency you need 500 × 0.04
= 20 requests in flight. A pool of 200 threads does not give more throughput: it gives
180 threads waiting and a hidden queue.
- Detecting an impossible measurement: if the generator says 1000 req/s with 10 ms of
latency and only holds 2 connections, the numbers do not add up (2 ≠ 10). One of the three
is mismeasured — usually latency, through coordinated omission.
- Knowing how much queue there really is: L grows when latency grows at constant
throughput. That is the operational definition of saturation.
6.2 Why latency explodes near 100 % utilisation
In the M/M/1 model (Poisson arrivals, exponential service, one server), with utilisation
ρ = λ/μ and service time S = 1/μ, mean response time is R = S / (1 − ρ). Direct arithmetic
consequence —it is not an empirical datum, it is the model:
| Utilisation ρ |
Response time R |
| 50 % |
2 × S |
| 80 % |
5 × S |
| 90 % |
10 × S |
| 95 % |
20 × S |
| 99 % |
100 × S |
What you have to take away, and it is what breaks almost every capacity plan:
- The relationship between utilisation and latency is not linear: it is an asymptote. The
last percentage points of utilisation are unbearably expensive in latency.
- A resource at 95 % is not "nearly fine": it is saturated. The margin is not slack, it is
the price of tail latency.
- Variability makes the curve worse. M/M/1 is the benign case; with service more variable
than exponential (the norm in real software: multimodal, with GC, with cache misses) latency
rises earlier and higher.
- You never size to a target utilisation without declaring the associated latency objective.
"Target 80 % CPU" without latency alongside it is half a decision. The specific target
utilisation number depends on the system and is derived from the latency budget: this skill
does not fix it (§8).
- Any target-utilisation table copied from a blog is suspect. The derivation is done with
your own measured S and your own p99 objective.
6.3 Saturation: what to look at, not what to assume
Utilisation hides saturation. You watch the queues, which is where latency lives: run queue
depth, I/O wait, connection pool queue, socket accept queue (backlog), block device queue
depth, cgroup throttling. A CPU at 60 % with a permanent run queue is saturated; the
utilisation percentage does not say so.
6.4 Concurrency, not capacity, is what has to be limited
- Admission limits are set on in-flight concurrency, not only on requests per second: that
is what protects against the asymptote in §6.2.
- Shedding load fast is better than queueing it: a queue that grows indefinitely turns a
spike into a total outage and also ruins the requests that could have been served. The design
of degradation and shedding belongs to
sre-practice-standards; the argument for why
belongs here.
- Retries: they multiply load exactly when the system is saturated.
6.5 Cache: it is a decision with a cost, not a patch
- A cache does not fix a bad algorithm: it hides it until the cache misses. Before caching,
you answer in writing: why is the thing you are about to cache expensive, and can it not be
made cheap?
- The real cost of a cache is invalidation and coherence, not memory. Every cache adds: an
ambiguous source of truth, a new failure mode (stale data served as good), and a cold-start
scenario in which the system does not withstand its nominal load.
- It is always declared: what invalidates each entry, how long stale data is tolerated, and
what happens when the cache is empty or down. If the answer to the last one is "everything
falls over", the cache is not an optimisation: it is a critical component without redundancy.
- A cache is measured by its effect on p99 latency, not by its hit rate. A 95 % hit rate
with the remaining 5 % at 2 s leaves an appalling p99.
- Simultaneous reload spikes (stampede) turn a cache miss into an outage. Mechanics and
mitigation:
caching-cdn-standards.
7. Sustainability and prohibitions
7.1 The record of the work
- Every optimisation is accompanied by a before and after measurement, in the same
environment, with the same script and with reported variance. Without both halves, the
change is reverted: there is no way to tell it apart from noise.
- Per-service performance register, versioned in the repository: current objective, dated
baseline, applied optimisations with their measurement, and a review date.
- Optimised code expires. Every non-obvious optimisation carries a comment with: what was
measured, how much it gained, over which version of which dependency, and a review date.
The reasons it expires are routine: the compiler or the JIT changes, the data volume changes,
the hardware changes, the library changes — and the optimisation goes from being an advantage
to being complexity nobody dares touch. An optimisation without a review date is debt with
interest.
- Premature optimisation — the criteria, not the quote. The rule is not "don't optimise": it
is that the cost of an optimisation is permanent complexity and its benefit is
hypothetical until measured. Therefore: you optimise when (a) there is a written objective
that is not being met, and (b) there is a measurement that points at that specific code.
Neither of the two conditions can be substituted by experience. Explicit exception:
decisions that are expensive to reverse — data schema, concurrency model, service
boundary, serialisation format — are reasoned with orders of magnitude before building.
That is not premature optimisation: it is design. Confusing the two is the lazy reading of
Knuth's sentence.
- Cadence: review the state of the profiling tools and of OTLP Profiles on the web
every quarter while the signal is still in alpha (§8). Review the performance objectives
when the data volume, the hardware or the topology changes — and at least once a year.
7.2 Explicit prohibitions
- ❌ Optimising without measuring first. Without a baseline there is no improvement: there
is an opinion with a
git commit.
- ❌ Declaring an improvement without a subsequent measurement in the same environment and
with the same methodology as the baseline.
- ❌ Deciding with the mean. And also forbidden is the chart that only shows the mean, and
the dashboard that averages percentiles across instances or sums them across services
(§3.2).
- ❌ Reporting percentiles obtained with a closed load generator against an open system, or
without declaring the arrival model. That is coordinated omission and the number is false
(§4.1).
- ❌
ab or the original wrk for measuring percentiles. Without CO correction, they are
good for "is it alive?" and nothing more.
- ❌ Publishing the result of a load test without declaring the environment parity, the
dataset and the hardware. And it is FORBIDDEN to present as production capacity a number
obtained in an environment that is not parity.
- ❌ Load testing against production without written authorisation, an agreed window and a
stop plan. It is a self-inflicted denial of service.
- ❌ Micro-optimising before having the time breakdown of the whole hot path (§5.1).
- ❌ Justifying an architecture decision with a microbenchmark (§4.4).
- ❌ Adding a cache to cover a slow query without hav
…(truncated)
1---2name: performance-engineering-standards3description: Use when a backend, runtime or system is slow and someone must prove why — defining a latency objective as percentile plus concurrency plus hardware, tail latency at p99 and p99.9, coordinated omission in load generators, open versus closed workload models, wrk2 -R, vegeta -rate, k6 constant-arrival-rate and ramping-arrival-rate executors, Gatling injectOpen versus injectClosed, JMeter Open Model Thread Group, Locust, HdrHistogram, the USE method (utilization, saturation, errors) and the RED method (rate, errors, duration), perf record and perf script, FlameGraph flamegraph.pl and stackcollapse, bpftrace and eBPF tracing, perf_event_paranoid, go tool pprof and net/http/pprof, py-spy, async-profiler and JFR, continuous profiling with Pyroscope, Parca or Grafana Profiles Drilldown, OTLP profiles and the OpenTelemetry eBPF profiler, sampling versus instrumentation overhead, CPU and allocation profiles, Amdahl's law, Little's law and queueing saturation near full utilization, latency budgets split across services4---56# Performance engineering standards78Criteria verified as of **August 2026**. Re-verify on the web before committing to anything (§8).910## 1. Scope and triggers1112Applies to **server-side performance methodology: system, runtime, service and the path that13joins them**. What is measured, with which statistic, with which load generator, how it is14profiled, in what order the culprits are hunted and what evidence is needed to accept that an15optimisation is an improvement and not a coincidence. **The thesis**: *measure before touching,16and know what to measure*. An optimisation without measurement before and after is not17engineering, it is superstition with a commit.1819Triggers: `perf record`/`perf report`/`perf script`/`perf stat`, `perf_event_paranoid`,20`flamegraph.pl`, `stackcollapse-perf.pl`, *flame graph*, *off-CPU*, `bpftrace`, eBPF, BCC,21`pprof`, `net/http/pprof`, `go tool pprof`, `py-spy`, `async-profiler`, JFR, `jemalloc` /22`heaptrack` / `massif`, Pyroscope, Parca, Profiles Drilldown, continuous profiling, OTLP23Profiles, `wrk`/`wrk2 -R`, `vegeta -rate`, k6 `constant-arrival-rate` / `ramping-arrival-rate`24/ VU, Gatling `injectOpen` / `injectClosed` / `rampUsersPerSec`, JMeter *Open Model Thread25Group*, Locust, `ab`, HdrHistogram, *coordinated omission*, *tail latency*, p95/p99/p99.9,26USE, RED, *golden signals*, Little's law, Amdahl's law, latency budget, load / stress / *soak*27/ spike testing, N+1, *cold start*, GC pause.2829**Not applicable**: see `web-performance-standards` (**the finest boundary in the catalogue,30and it is already written from the other side**: theirs are the **browser and the real user**31—Core Web Vitals, RUM at p75, *bundle* budget, hydration cost, CLS—; **here the server, the32runtime and the system**. The contact point is **TTFB**: *it is their input metric and my33output metric* — **how server response time is reduced belongs here; that this TTFB feeds into34LCP and how much it weighs in the field p75 is theirs**. Operational corollary: an impeccable35server p99 does not contradict a slow website, and vice versa; **a debate about one is never36closed with the other's number**), `sre-practice-standards` (**important boundary**: SLO, SLI,37*error budget*, *burn rate* alerts, capacity planning and production reliability are theirs.38Rule: ***"how much latency can we afford and what do we do if we exceed it?" belongs to39`sre-practice`; "why is it slow and what fixes it?" belongs here***. The SLO defines the40objective; this skill provides the method that meets it), `observability-standards` (**the41telemetry platform is theirs**: collectors, OTel Collector, storage, retention, trace sampling,42cardinality, dashboards and alerts — **continuous profiling is ingested and stored there**;43**here what is profiled, at what sampling frequency and how a *flame graph* is read**),44`testing-qa-standards` (**reciprocal declared in both directions**: there the load test is45**planned** as a test type —scenarios, data, stopping criteria, and the rule that it never goes46in the PR gate—; **here the load model (open or closed), the measurement methodology and the47interpretation of the result**, §4. A load script written there that uses a closed model48against an open system produces an invalid number according to this skill),49`data-platform-standards`, `sql-standards`, `mysql-mariadb-dba-standards`,50`oracle-dba-standards`, `sqlserver-dba-standards`, `nosql-standards`, `timeseries-db-standards`51and `search-engines-standards` (**engine, index and query plan tuning are theirs**; here only52the methodology that proves the bottleneck is in the database and with what evidence you knock53on their door), `python-standards`, `go-standards`, `rust-standards`,54`jvm-spring-standards`, `dotnet-standards`, `cpp-standards`, `c-standards`,55`typescript-standards` and the other language skills (**the runtime's specific profiler, its56flags and its garbage collector tuning are theirs**; here the common method and when to reach57for them), `gpu-computing-standards` (kernels, occupancy and GPU memory),58`caching-cdn-standards` (HTTP cache policy, the CDN and purging; **here the cache as an59architecture decision with a coherence cost**, §6.5), `networking-standards` and60`network-troubleshooting-standards` (**network diagnosis is theirs**: RTT, loss, MTU,61*retransmits*, capture; here only placing the network in the suspect tree and handing them the62case), `kubernetes-standards` (*requests*/*limits*, HPA, cgroup CPU *throttling*),63`linux-administration-standards` and `linux-storage-standards` (system and storage tuning),64`operating-systems-standards` (***"how is it measured?" belongs here; "what OS mechanism65produces that number?" is theirs*** — scheduler, memory management, the real cost of a system66call, `fsync` guarantees), `finops-standards` (**cost as a metric and its budget**; here cost67only appears as an argument for sizing) and `microservices-architecture-standards` (the68division of responsibilities between services; here the division of the **latency budget**69between them, §3.3), `refactoring-tech-debt-standards` (**optimising is not refactoring**:70refactoring changes the structure **without** changing observable behaviour; optimising changes71an observable characteristic —latency— and therefore **is measured before and after and is72justified with that data**. Debt criteria, its register and its funding are theirs; the73methodology of finding and proving the improvement, here. And the warning they share:74**optimised code expires** and needs a review date, whereas a well-done refactor does not),75`green-it-standards` (**the methodology of measuring before touching,76profiling and proving the improvement belongs here**; **the energy and carbon metric is77theirs**, with its own accounting and its own traps. Warning both share: **code efficiency and78footprint reduction are not the same thing** — the real order of impact starts with switching79off what is idle and right-sizing, and optimising code for sustainability is only justified at80a scale that has to be calculated, not inherited).8182## 2. Default decisions8384> Verify the latest version and the licence on the web before pinning it in a real project (§8).8586### 2.1 The two named methods, cited at source8788They are not styles: they are **checklists with an author, and they apply to different89things**. Using them crosswise is the most expensive framing error.9091**USE — Brendan Gregg** (`brendangregg.com/usemethod.html`; published on his dtrace.org blog92on 29 Feb 2012 and in ACM Queue as *Thinking Methodically about Performance*, 2012, and in93CACM, 2013). Verbatim:9495> *"For every resource, check utilization, saturation, and errors."*96> *"resource: all physical server functional components (CPUs, disks, busses, ...)"*97> *"utilization: the average time that the resource was busy servicing work"*98> *"saturation: the degree to which the resource has extra work which it can't service, often queued"*99> *"errors: the count of error events"*100101Scope declared by the author himself, verbatim: *"It solves about 80% of server issues with 5%102of the effort"*, and its limit, also verbatim: *"There are many problem types it doesn't103solve, which will require other methods and longer time spans"* / *"While the USE Method may104find 80% of server issues, latency-based methodologies (eg, Method R) can approach finding 100%105of all issues."* **The 80/5 is cited as what it is —a claim by the author, not a study— or it106is not cited at all.**107108**RED — Tom Wilkie**, created **in 2015** (Grafana Labs, *The RED Method: How to Instrument Your109Services*, presented at GrafanaCon EU 2018; Wilkie arrived at Grafana with the acquisition of110Kausal, and developed RED at Weaveworks). Verbatim from the definition:111112> *"Rate (the number of requests per second), Errors (the number of those requests that are113> failing), Duration (the amount of time those requests take)"*114115Declared reason, verbatim: *"The USE Method doesn't really apply to services; it applies to116hardware, network disks, things like this. We really wanted a microservices-oriented monitoring117philosophy, so we came up with the RED Method."* And the relationship with the two neighbours,118verbatim: on the *Four Golden Signals* (latency, traffic, errors, saturation), *"This is119basically the same as the RED Method, but includes saturation"*; and on using them together,120*"the RED Method is about caring about your users and how happy they are... and the USE Method121is about caring about your machines and how happy they are... They're complimentary."*122123⚠ **Discrepancy declared in the source itself**: the Grafana article heads **both** lists with124*"For every resource, monitor:"*, including RED's, despite the text immediately before it125arguing that RED exists precisely because USE does **not** apply to services. The formulation126consistent with the rest of the article and with universal usage is **"for every *service*"**.127When citing RED, cite the triad, not the heading.128129**Rule of use**: **USE for resources** (CPU, memory, disk, network, controllers, cgroups) and130**RED for services and endpoints**. In an incident you walk **both**: RED says *which service*131hurts, USE says *which resource* causes it. The SRE Book's *golden signals* are RED +132saturation and belong to `sre-practice-standards`; **here they are used as a search index, not133as an alert**.134135### 2.2 Tools — status and licence verified as of Aug 2026136137| Use | Default | Licence / verified status | Note |138|---|---|---|---|139| CPU profiling on Linux | **`perf`** | GPL-2.0 (`SPDX-License-Identifier: GPL-2.0` in `tools/perf` of the kernel tree) | Sampling, in the kernel; no third-party dependencies |140| Profile visualisation | **FlameGraph** (`stackcollapse-*.pl` + `flamegraph.pl`) | ⚠ **CDDL** — *"all files in this distribution are released under the Common Development and Distribution License (CDDL)"* (`docs/cddl1.txt`) | **It is not MIT nor Apache**: per-file copyleft. If it gets packaged, it goes through review |141| Ad hoc dynamic tracing | **`bpftrace`** | Apache-2.0 (raw `LICENSE`) — v0.26.1 (2 Jun 2026) | For questions no counter answers |142| Continuous profiling (OSS) | **Grafana Pyroscope** | ⚠ **AGPL-3.0** (raw `LICENSE`) — active weekly releases (Aug 2026) | AGPL: relevant if offered as a service. UI: *Profiles Drilldown* (plugin, **public preview**, not GA) |143| Continuous profiling (alternative) | **Parca** | Apache-2.0 (raw `LICENSE`) — active, engine behind Polar Signals Cloud | Permissive licence; smaller ecosystem |144| Interchange standard | **OTLP Profiles** | ⚠ **Public alpha since 26 Mar 2026** | §2.4. Do **not** bet critical production on it yet |145| Constant-rate load | **`wrk2`** (`-R` mandatory) | Apache-2.0 (raw `LICENSE`) | Historical reference for corrected latency; **old repo, verify before adopting** |146| Programmable load | **k6** with `constant-arrival-rate` / `ramping-arrival-rate` executor | ⚠ **AGPL-3.0** (raw `LICENSE.md`) — v2.1.0 (30 Jun 2026) | The VU default is a **closed model**: the executor has to be chosen by hand (§4.2) |147| JVM load | **Gatling** (`injectOpen`) | Apache-2.0 (`LICENSE.txt`; *"Gatling Open Source is licensed under Apache 2.0"*) — v3.15.1 (25 May 2026) | Gatling Enterprise is a separate paid product |148| Legacy GUI load | JMeter | Apache-2.0 — **5.6.3, no new release since Jan 2024** | *Open Model Thread Group* declared **experimental** (§4.2). It is not the default for anything new |149| Simple HTTP attack | `vegeta` (`-rate`) | MIT (raw `LICENSE`) | Good for one endpoint, not for a user journey |150| Load in Python | Locust | MIT (raw `LICENSE`) | Requires explicitly choosing the arrival model |151| Latency distribution | **HdrHistogram** | — | Latency recording is **always** a histogram, never a running average (§3.2) |152153**Warnings that correct common assumptions**:154- **k6 is AGPL-3.0**, not Apache or MIT. If it is embedded in a product or offered as a155 service, that is a legal decision, not a tool choice.156- **Pyroscope is AGPL-3.0**; **Parca is Apache-2.0**. They are the same category with licences157 incompatible in purpose: the choice is made with the lawyer present, not by the UI.158- **FlameGraph is CDDL**, not permissive. It is the case most people assume is MIT.159- **JMeter has gone more than two years without a release** on its official site (5.6.3, Jan160 2024). The GitHub feed agrees here, but **the GitHub feed is not the source of truth**: Apache161 publishes at `jmeter.apache.org`. Check there (§8).162163### 2.3 Sampling versus instrumentation164165| | Sampling | Instrumentation (*tracing*/*instrumenting*) |166|---|---|---|167| What it does | Interrupts at a fixed frequency and records the stack | Records every event that has been marked |168| Bias | Statistical: misses the rare, sees the expensive well | Observer bias: the cost grows with the event's frequency |169| Cost | Bounded and predictable by frequency | **Unbounded**: an instrumented hot function can dominate its own profile |170| Use | **By default**, and the only acceptable one in continuous production | Ad hoc, bounded, on the already suspect path |171| Trap | Does not see **off-CPU** time (I/O wait, blocking, scheduler) | A counter per call turns the cheap into the expensive and **changes the result it was going to measure** |172173**Hard rules**:174- **In production you profile by sampling.** Fine-grained instrumentation in production only175 with bounded scope, a time window and automatic deactivation.176- **Frequency by convention: non-round values** (e.g. `perf record -F 99` instead of177 100 Hz). A frequency that coincides with a timer, a `tick` or a system loop always samples178 the same phase and produces a biased profile that looks perfect.179- **A CPU profile does not explain a waiting latency.** If the service is slow but the CPU is180 idle, the CPU profile will come out empty of culprits: you need an **off-CPU** profile181 (blocking, scheduler, I/O) or traces. It is the most frequent diagnostic error after182 *coordinated omission*.183- **A *flame graph* is read by width, not by height.** Height is stack depth; width is time. A184 tall, narrow stack costs nothing.185- **Profiling CPU and profiling memory allocation are two different questions.** The186 allocation/*heap* profile is the one that explains GC pressure and many tail latencies; the187 CPU one does not see it. Both, or the diagnosis is incomplete.188189### 2.4 OpenTelemetry Profiles — verified status, and why it matters190191**Status: public alpha since 26 Mar 2026** (official OpenTelemetry announcement, *OpenTelemetry192Profiles Enters Public Alpha*, signed by the Profiling SIG). What is verifiable from it:193194- Format with stack deduplication and dictionary tables; optional link to `trace_id` /195 `span_id` for **cross-signal correlation** (profile ↔ trace); on-wire size reduction cited by196 the announcement itself as **"40% smaller wire size"** for the string dictionary.197- Relationship with `pprof`, verbatim: *"Originally inspired by the pprof format and developed in198 collaboration with pprof maintainers, OTLP Profiles has evolved into an independent standard...199 Data in the original pprof format can be round-trip converted to/from OTLP Profiles with no200 loss of information."*201- The **whole-system eBPF profiler donated by Elastic** is already an official Collector202 component (receiver), with on-target symbolisation for Go, ARM64 for Node.js/V8, initial203 BEAM and .NET 9/10 support.204- Warning from the announcement itself, verbatim: *"As the signal is still under development,205 production-ready backends have not yet emerged but multiple vendors are working on supporting206 OpenTelemetry Profiles."*207208**Criteria**: **alpha means alpha.** OTLP Profiles is adopted as the **target format** when209designing today (it avoids getting tied to a proprietary agent), but **the production backend210is still Pyroscope or Parca**, and the architectural commitment is reviewed at each release211(§8). Pinning a critical backend today on the alpha signal is debt with a due date.212213## 3. The prior discipline: define the objective before measuring214215### 3.1 A performance objective without these four pieces is not an objective216217**Metric + percentile + load + hardware.** Miss one and the number is not comparable with218anything, not even with itself the following week.219220```221p99 latency of /api/orders ≤ 300 ms222 at 500 req/s sustained (open arrival model, Poisson)223 on 3 replicas of 2 vCPU / 4 GiB, database with the production dataset224 measured at the client, not at the server225```226227- **"Fast" is not an objective.** Neither is "under 300 ms", if it does not say at which228 percentile nor under how much load: any system meets any latency with one user.229- **The objective is expressed in user terms**, not component terms: "search responds in X",230 not "the query takes Y". The per-component budget is **derived** from the objective (§3.3),231 never the other way round.232- **It is measured at the client.** The time the server thinks it took excludes accept233 queueing, TLS, and precisely the queue that causes the problem.234235### 3.2 Why the mean lies, and what is used instead236237- **The latency distribution is not normal**: it has a long right tail and is usually238 multimodal (cache hit / cache miss; fast path / slow path; with GC / without GC).239 **On a multimodal distribution the mean can fall in a valley where there is no real240 request at all.** A mean of 120 ms is compatible with "all of them take 120 ms" and with "90 %241 take 20 ms and 10 % take a second": they are two different systems and only one is acceptable.242- **The mean is insensitive to the tail by construction**; the tail is exactly what the user243 remembers and what saturates the system upstream.244- **The median lies just the same, more discreetly.** Decisions are made with **p99** as a245 minimum, and with **p99.9** in services with internal fan-out (§3.4).246- **Percentiles are not averaged and are not summed.** The mean of the p99s of ten instances is247 not the p99 of the set; summing the p99 of three dependencies does not give the p99 of the248 total. You aggregate with histograms (HdrHistogram, `histogram_quantile` over *buckets*),249 never with arithmetic over already-computed quantiles. A dashboard that averages percentiles250 is broken however pretty it looks.251- **Standard deviation over a long-tailed distribution means nothing.** It is not reported.252- **Maximum and p100**: they are recorded (they expose the worst case and pathological253 *outliers*), not used as an objective — a single event sets the number.254255### 3.3 Latency budget and its division256257The user objective is **divided** among the stages of the path, and **the sum of the budgets is258less than the objective**, not equal to it: the margin pays for retries, variance and what does259not exist yet.260261- The budget is written **per dependency**, with a named owner, in the repository.262- **A service cannot promise a p99 better than the worst p99 on its critical path.** Before263 committing to a number, the floor of the synchronous dependencies is summed.264- **Every added synchronous hop consumes budget permanently.** Adding a network call to a hot265 path is an architecture decision with a price; the price is written in the ADR266 (`microservices-architecture-standards`).267- **Retries multiply load at the worst possible moment.** A retry without *backoff* with268 *jitter* and without a retry budget turns a degradation into a collapse. The pattern belongs269 to reliability (`sre-practice-standards`); **the effect on tail latency belongs here**: the270 retry is the user's time.271- **Timeouts are derived from the budget, they are not copied.** A client *timeout* larger than272 the stage's budget is a decorative *timeout*.273274### 3.4 Tail amplification: why the internal p99 is the user's p50275276If a user request opens **N** internal calls in parallel and waits for all of them, the277probability that **none** falls into the slow tail decays with N. With N=100 independent calls278and an internal p99, the probability that all 100 are below the p99 is2790.99¹⁰⁰ ≈ **0.366**: that is, **~63 % of user requests touch at least one call above the280internal p99**. It is arithmetic, not a study.281282Operational consequences, and they are harsh:283- **In fan-out services you optimise p99.9, not p99.** The internal p99 is already the typical284 user case.285- **Reducing fan-out is a first-order latency optimisation**, often bigger than any286 micro-optimisation of code.287- **A single slow component on the path dominates the result** however fast the rest are. See288 §5.1 (Amdahl).289290## 4. Measuring: load tests that produce a valid number291292### 4.1 Coordinated omission — the most expensive and least known measurement error293294**What it is.** Term coined by **Gil Tene**. The load generator **inadvertently coordinates295with the measured system**: if the next send depends on the previous one having finished, when296the system stalls **the generator stops sending** for exactly the bad interval. Result: instead297of the N slow requests a real user would have suffered, the histogram records **a single** slow298sample. High percentiles come out **orders of magnitude** better than they were.299300**Why it is so toxic**: it is not a failure of the measured system, it is a failure of the one301measuring, and **it gets worse exactly when the system gets worse** — that is, the tool lies to302you more the more the truth matters. A closed generator can report an excellent p99 for a303service that stops for whole seconds.304305**Canonical illustration** (mechanics, not an empirical figure): target 10 req/s, normal306service of 50 ms; the system stalls for 5 s. A closed generator records **one** request of307~5 s and none of the ~50 it should have sent in that gap. "Corrected" latency also counts the308time each pending request **should** have been waiting since its expected arrival instant.309310**What avoids it, verified**:311312| Tool | Model | Verified status |313|---|---|---|314| **`wrk2`** | Mandatory constant rate (`-R`), HdrHistogram, reports **corrected** and **uncorrected** latency | Designed by Tene specifically against CO |315| **k6** | `constant-arrival-rate` / `ramping-arrival-rate` = open; **VU in a loop = closed** | Its own documentation names it, verbatim: *"In some testing literature, this problem is known as coordinated omission"* — describing its **closed model** |316| **Gatling** | `injectOpen` (open, `rampUsersPerSec`) vs. `injectClosed` (closed); **they cannot be mixed in one scenario** | Explicit choice in the script |317| **`vegeta`** | `-rate` = open by design | One endpoint, not a user journey |318| **Locust** | Requires a choice; its documentation warns that a test which does not reach the throughput target shows artificially low response times | Explicit config |319| **JMeter** | *Open Model Thread Group*, verbatim from its documentation: ***"This thread group is experimental, and it might change in the future releases."*** | The classic *Thread Group*s are **closed** |320| **`ab`, `wrk` (original)** | Closed, no correction | **Banned** for measuring percentiles (§7) |321322**Hard rules**:323- **The arrival model is declared in writing in the test report.** A result without a declared324 open or closed model is a number without units.325- **Open system → open generator.** A public API, a website, a database with independent326 clients are open systems: real users **do not wait** for the ones next to them to finish327 before clicking. Measuring them with a closed generator is measuring a different system.328- **The closed model is legitimate** when the real system *is* closed: a job queue with N fixed329 workers, a batch, an internal client with bounded concurrency. The rule is not "always open",330 it is **"the generator's model must be the system's model"**.331- **If the generator hits its own limits** (CPU, sockets, descriptors, a single node against a332 cluster), the test is **invalid and is discarded** — it is not interpreted. The generator is333 monitored with the same seriousness as the target.334- When using `wrk2`, **the corrected latency** is reported. Publishing the uncorrected one335 without saying so is publishing the error.336337### 4.2 The four test types, and what each one answers338339They are planned in `testing-qa-standards`; **the interpretation belongs here**.340341| Type | Question | Duration | Failure signal |342|---|---|---|---|343| **Load** | Does it meet the objective under the expected load? | Enough to stabilise | The latency budget is exceeded |344| **Stress** | Where is the breaking point and **how** does it break? | Ramp until degradation | It breaks in an uncontrolled way (no *shed*, no degradation, with cascade) |345| **Endurance (*soak*)** | Does it degrade over time? | Hours or days | Growing drift in memory, descriptors, connections, latency; fragmentation; poisoned cache |346| **Spike** | Does it survive an abrupt jump and recover? | Minutes | It does not return to the previous level after the spike — the most expensive failure and the least tested |347348**The *soak* is the one most people skip and the one that prevents the most incidents**: memory349and descriptor leaks do not show up in a ten-minute test, by definition.350351### 4.3 Validity requirements for a test352353Miss **one** and the result is not published:3543551. **Declared environment parity.** **A load test against an environment that is not356 production parity does not produce a number: it produces a useless number.** The factors357 that break it and are not negotiable: **data volume and distribution** (an index over a358 thousand rows always fits in memory and is always fast; the query plan can be *another one*359 at real volume), **topology** (one replica versus three, with or without a real load balancer360 and real proxies), **machine class and noisy neighbours**, **real network latency between361 components**, **cgroup limits / CPU *throttling***, and **identical runtime configuration**362 (GC, *pools*, *timeouts*). If there is no parity, the result is only good for **comparing363 with itself** across identical runs, and it is written that way in the report.3642. **Warm-up and discard.** JIT, caches, connection *pools*, page tables and filesystem caches365 need a warm-up period **that is discarded from the result**. Including start-up in the366 histogram contaminates the tail with an artifact.3673. **Representative data with realistic cardinality.** A single user, a single key or a single368 product turns the test into a cache test. The access distribution matters as much as the369 volume (Zipf, not uniform).3704. **Duration long enough to reach steady state** and for the periodic events to occur: major371 GC, connection rotation, cache expiry, scheduled tasks, *compaction*.3725. **Repeatability**: N≥3 runs. If the variance between runs exceeds the difference you want to373 demonstrate, **nothing has been demonstrated**.3746. **System under test observed during the test**: USE on every resource, RED on every375 service. A test that only produces the final number does not allow diagnosing anything.3767. **Reproducible record**: artifact version, *commit*, configuration, dataset, tool, script,377 arrival model and hardware. **A result nobody can reproduce is an anecdote.**378379### 4.4 Microbenchmarks380381- **A microbenchmark measures the microbenchmark.** It is valid for comparing two382 implementations of the same function, and **not** for predicting the service's latency.383- The language's harness is used (it is theirs: JMH, `go test -bench`, `criterion`,384 `pytest-benchmark`), with warm-up and statistical significance. **Without reported variance,385 it is not accepted.**386- **The compiler can eliminate the measured code.** If the result is not consumed, the387 optimiser deletes it and you measure an empty loop. It is a silent failure.388- **Justifying an architecture change with a microbenchmark is forbidden** (§7).389390## 5. The right order of the search391392### 5.1 Amdahl's law: the ceiling is set by what you do not optimise393394Amdahl (1967, *AFIPS Conference Proceedings* vol. 30, pp. 483-485 — verify the reference395before citing it in a formal document, §8). Formulation: if a fraction **p** of the time is396sped up by a factor **s**, the total improvement is **1 / ((1−p) + p/s)**.397398Consequence that decides the order of work: **making infinitely fast a part that takes 5 % of399the time produces, at most, a 5.3 % improvement.** Therefore:4004011. **Measure the whole hot path before touching anything.** The distribution of time by stage402 is the first artifact, always.4032. **Optimise in order of time fraction, not in order of ease or of interest.**4043. **Recalculate after each change**: once the bottleneck is removed, the bottleneck is another405 one and the priority list changes entirely. A performance task list **expires as soon as the406 first item is completed**.4074. **Scaling out does not fix what is serialised.** That is the practical corollary: if 20 % of408 the work is serial (a global lock, a hot table, a single service), doubling replicas does not409 give 2×. Gunther's Universal Scalability Law adds the **coherence** term (the cost of the410 replicas agreeing with each other), which can make **adding capacity make performance411 worse**. Verify the formulation before citing it (§8).412413### 5.2 The suspect tree, by layer and in this order414415You walk it top-down, and **each layer is ruled out with evidence**, not with intuition:4164171. **Unnecessary work** — the optimisation that always wins. Is something being computed that418 nobody looks at? Is more being requested than is used? Is a whole object being serialised to419 read one field? It is removed; it is not optimised.4202. **Application**: algorithmic complexity over the **real** input size; **N+1** of queries or421 of remote calls (the classic and the most expensive); serialisation and deserialisation422 (often the biggest CPU consumer in a "network" service); data copies; locks and contention;423 synchronous work that should have been asynchronous.4243. **Runtime**: garbage collector pauses and pressure, *heap* size and policy, badly sized425 *pools* (connections, threads), cold start, JIT. **The specific tuning belongs to the426 language skill**; here only the evidence that it is there.4274. **Database**: query without an index, changed plan, locks, *pool* exhaustion, long428 transactions, `N+1`. **You knock on the engine skill's door with evidence**: the query, its429 plan and its share of total time.4305. **Network**: RTT, hops, MTU, loss and retransmission, TLS handshake, DNS.431 **`network-troubleshooting-standards` diagnoses**; here you only locate it and hand it over.4326. **Disk / storage**: service latency, queue depth, random vs. sequential pattern, `fsync`,433 IOPS saturation.4347. **Kernel and platform**: scheduler, cgroup *throttling* (the favourite cause of bad p99 on435 Kubernetes with aggressive CPU *limits*), NUMA, descriptor exhaustion, conntrack table,436 memory pressure.437438**Cross-cutting rule**: **a layer is ruled out with a measurement, not with an argument.** "It439can't be the database" does not rule out the database.440441### 5.3 Latency versus throughput: improving one worsens the other442443They are not the same quantity and **they are optimised with opposing techniques**:444445| Technique | Effect on throughput | Effect on latency |446|---|---|---|447| Batching | ↑↑ | ↑ (waits to fill the batch) |448| Deeper queue | ↑ (absorbs spikes) | ↑↑ (more waiting time) |449| More concurrency | ↑ until saturation | ↑ past saturation |450| Compression | ↑ if the network is the bottleneck | ↑ CPU at both ends |451| Speculative execution / *hedged* requests | ↓ (duplicated work) | ↓↓ in the tail |452453**Hard rule**: **which of the two is being optimised is declared before starting.** One team454optimising throughput and another optimising latency on the same system undo each other's work.455And **a bigger queue never fixes a capacity problem**: it only turns fast errors into long456waits, which is worse.457458## 6. Queues, saturation and sizing459460### 6.1 Little's law461462**J. D. C. Little (1961), *A Proof for the Queuing Formula: L = λW*, Operations Research4639(3):383-387.** L = mean number of units in the system, λ = mean arrival rate, W = mean time in464the system. It is **independent of the distribution** of arrivals, of service and of the queue465discipline: that is why it can be applied almost always.466467Useful form in systems: **concurrency = throughput × latency**. Three calculations come out of468that, and they are done **before** touching configuration:469470- **Sizing a *pool***: to serve 500 req/s with 40 ms of latency you need 500 × 0.04471 = **20** requests in flight. A *pool* of 200 threads does not give more throughput: it gives472 180 threads waiting and a hidden queue.473- **Detecting an impossible measurement**: if the generator says 1000 req/s with 10 ms of474 latency and only holds 2 connections, the numbers do not add up (2 ≠ 10). **One of the three475 is mismeasured** — usually latency, through *coordinated omission*.476- **Knowing how much queue there really is**: L grows when latency grows at constant477 throughput. That is the operational definition of saturation.478479### 6.2 Why latency explodes near 100 % utilisation480481In the **M/M/1** model (Poisson arrivals, exponential service, one server), with utilisation482ρ = λ/μ and service time S = 1/μ, mean response time is **R = S / (1 − ρ)**. Direct arithmetic483consequence —**it is not an empirical datum, it is the model**:484485| Utilisation ρ | Response time R |486|---|---|487| 50 % | 2 × S |488| 80 % | 5 × S |489| 90 % | 10 × S |490| 95 % | 20 × S |491| 99 % | 100 × S |492493What you have to take away, and it is what breaks almost every capacity plan:494495- **The relationship between utilisation and latency is not linear: it is an asymptote.** The496 last percentage points of utilisation are unbearably expensive in latency.497- **A resource at 95 % is not "nearly fine": it is saturated.** The margin is not slack, it is498 the price of tail latency.499- **Variability makes the curve worse.** M/M/1 is the *benign* case; with service more variable500 than exponential (the norm in real software: multimodal, with GC, with cache misses) latency501 rises earlier and higher.502- **You never size to a target utilisation without declaring the associated latency objective.**503 "Target 80 % CPU" without latency alongside it is half a decision. The **specific target504 utilisation number depends on the system and is derived from the latency budget**: this skill505 does not fix it (§8).506- **Any target-utilisation table copied from a blog is suspect.** The derivation is done with507 your own measured S and your own p99 objective.508509### 6.3 Saturation: what to look at, not what to assume510511Utilisation hides saturation. You watch **the queues**, which is where latency lives: run queue512depth, I/O wait, connection *pool* queue, socket accept queue (*backlog*), block device queue513depth, cgroup *throttling*. **A CPU at 60 % with a permanent run queue is saturated**; the514utilisation percentage does not say so.515516### 6.4 Concurrency, not capacity, is what has to be limited517518- **Admission limits are set on in-flight concurrency**, not only on requests per second: that519 is what protects against the asymptote in §6.2.520- **Shedding load fast is better than queueing it**: a queue that grows indefinitely turns a521 spike into a total outage and also ruins the requests that could have been served. The design522 of degradation and *shedding* belongs to `sre-practice-standards`; **the argument for why523 belongs here**.524- **Retries**: they multiply load exactly when the system is saturated.525526### 6.5 Cache: it is a decision with a cost, not a patch527528- **A cache does not fix a bad algorithm: it hides it until the cache misses.** Before caching,529 you answer in writing: why is the thing you are about to cache expensive, and can it not be530 made cheap?531- **The real cost of a cache is invalidation and coherence**, not memory. Every cache adds: an532 ambiguous source of truth, a new failure mode (stale data served as good), and a cold-start533 scenario in which the system does **not** withstand its nominal load.534- **It is always declared**: what invalidates each entry, how long stale data is tolerated, and535 **what happens when the cache is empty or down**. If the answer to the last one is "everything536 falls over", the cache is not an optimisation: it is a critical component without redundancy.537- **A cache is measured by its effect on p99 latency, not by its hit rate.** A 95 % hit rate538 with the remaining 5 % at 2 s leaves an appalling p99.539- **Simultaneous reload spikes (*stampede*)** turn a cache miss into an outage. Mechanics and540 mitigation: `caching-cdn-standards`.541542## 7. Sustainability and prohibitions543544### 7.1 The record of the work545546- **Every optimisation is accompanied by a before and after measurement, in the same547 environment, with the same script and with reported variance.** Without both halves, the548 change is reverted: there is no way to tell it apart from noise.549- **Per-service performance register**, versioned in the repository: current objective, dated550 baseline, applied optimisations with their measurement, and a **review date**.551- **Optimised code expires.** Every non-obvious optimisation carries a comment with: what was552 measured, how much it gained, over which version of which dependency, and a **review date**.553 The reasons it expires are routine: the compiler or the JIT changes, the data volume changes,554 the hardware changes, the library changes — and the optimisation goes from being an advantage555 to being complexity nobody dares touch. **An optimisation without a review date is debt with556 interest.**557- **Premature optimisation — the criteria, not the quote.** The rule is not "don't optimise": it558 is that the cost of an optimisation is **permanent complexity** and its benefit is559 **hypothetical until measured**. Therefore: you optimise when (a) there is a written objective560 that is not being met, and (b) there is a measurement that points at that specific code.561 Neither of the two conditions can be substituted by experience. **Explicit exception**:562 decisions that are **expensive to reverse** — data schema, concurrency model, service563 boundary, serialisation format — are reasoned with orders of magnitude **before** building.564 That is not premature optimisation: it is design. Confusing the two is the lazy reading of565 Knuth's sentence.566- **Cadence**: review the state of the profiling tools and of OTLP Profiles on the web567 **every quarter** while the signal is still in alpha (§8). Review the performance objectives568 when the data volume, the hardware or the topology changes — and at least once a year.569570### 7.2 Explicit prohibitions571572- ❌ **Optimising without measuring first.** Without a baseline there is no improvement: there573 is an opinion with a `git commit`.574- ❌ **Declaring an improvement without a subsequent measurement** in the same environment and575 with the same methodology as the baseline.576- ❌ **Deciding with the mean.** And also forbidden is the chart that only shows the mean, and577 the dashboard that **averages percentiles** across instances or sums them across services578 (§3.2).579- ❌ **Reporting percentiles obtained with a closed load generator against an open system**, or580 without declaring the arrival model. That is *coordinated omission* and the number is false581 (§4.1).582- ❌ **`ab` or the original `wrk` for measuring percentiles.** Without CO correction, they are583 good for "is it alive?" and nothing more.584- ❌ **Publishing the result of a load test without declaring the environment parity**, the585 dataset and the hardware. And it is **FORBIDDEN** to present as production capacity a number586 obtained in an environment that is not parity.587- ❌ **Load testing against production without written authorisation, an agreed window and a588 stop plan.** It is a self-inflicted denial of service.589- ❌ **Micro-optimising before having the time breakdown of the whole hot path** (§5.1).590- ❌ **Justifying an architecture decision with a microbenchmark** (§4.4).591- ❌ **Adding a cache to cover a slow query** without hav592593…(truncated)