Latency Statistics

Latency Statistics

Purpose

Make latency numbers answer a stated decision. This skill prevents a confident but underidentified result: timeouts omitted from a “successful-request p99”, a fleet quantile fabricated by averaging instance quantiles, or millions of correlated requests presented as millions of independent replications.

Latency is a distribution. Every rule here follows from that one fact.

Workflow

1. Define the estimand. Name population/cohort, start and end clock, success/error/timeout treatment, quantile definition, observation window and grouping. “Endpoint p99” is incomplete until these are fixed.
2. Preserve denominator and missingness. Report offered, admitted, completed, failed, cancelled, timed-out, dropped and still-in-flight counts. Time to an observed timeout response is a terminal duration; time to an unobserved eventual completion may instead be right-censored. State which variable is being estimated; excluding timeouts selects a different, success-only population.
3. Select statistics by decision. Quantiles answer threshold/tail questions; the mean answers expected latency for that population, not CPU/service demand unless that is what the clock measures. Threshold fractions directly answer “what proportion met 300 ms?”. Include counts and uncertainty. Do not prescribe the same p50/p90/p99 set for every decision.
4. Inspect distribution and time. Use histograms or empirical CDFs plus a time view. A single p99 cannot distinguish modes, queue growth, a periodic pause or a small failed cohort.
5. Check representational error. Record units, range, bucket layout, overflow/saturation, quantisation and rolling-window semantics. A statistically precise estimate of a coarse or truncated histogram is still wrong.
6. Aggregate mergeable distributions before querying. Add compatible histogram counts or raw observations, then compute the quantile. Never average instance/window quantiles.
7. Audit the observation process. For load generators, compare scheduled/offered/started/ completed work and inspect generator saturation; determine whether response completion controls future issue times. See references/coordinated-omission.md.
8. Compare treatments at the independent level. Define practical effect, experimental unit and pairing/blocking; estimate the treatment contrast with uncertainty. Requests inside one run are not automatically independent replications. See references/comparing-two-measurements.md.

Rules

• The statistical rules have no Java baseline. Before implementing the partial instrumentation examples, inspect the project's JDK, resolved HdrHistogram/Micrometer versions and registry/ backend support; adopting this skill does not authorize upgrades.
• A sample quantile exists even for small n, but may be almost entirely determined by the largest observations and have wide population-quantile uncertainty. Record n, the estimator/interpolation rule, and an interval or rank bounds; never relabel it “undefined”.
• n(1−p) is a useful tail-resolution diagnostic, not a universal minimum. Required sample size depends on desired value/rank precision, local density, dependence, censoring and the decision's error costs.
• A mean can exceed p99 when fewer than 1% of observations are extremely large. That is useful evidence about total cost and an unreported extreme tail—not evidence that means are useless.
• Standard deviation does not assume normality. It may be unstable or hard to interpret for a heavy tail, so accompany it with robust/distributional summaries; do not discard it by dogma.
• A maximum is meaningful for bounded-window forensics and safety limits but is highly sample-size-dependent. Do not substitute a more extreme quantile without enough resolution; state the operational question and estimator.
• A test of means does not test quantiles. Normality is not the central defect; wrong estimand, dependence, hierarchy, nonstationarity and informative missingness are. Choose analysis from the contrast, not from a blanket ban on a named test.
• Investigate the temporal pattern before the amplitude: a p99.9 spike every 30 minutes suggests a periodic mechanism, but does not identify one. Correlate event-level evidence; cache TTLs, jobs, traffic and metric windows can share the same period.
• Fan-out tail probability depends on correlation, retries, hedging and which branches lie on the critical path. 1−(1−q)^k applies only to independent branches with identical slow-event probability q; union bounds and measured joint behaviour are safer than “three times 1%”.

Required decision artifact

Population/cohort:  route, region, outcome policy, offered/admitted/completed counts
Clock:              start event → terminal event; monotonic clock and units
Window/state:       cold/ramp/sustained; exact interval and load
Representation:     raw/HDR/classic/native; range, buckets, precision, overflow
Estimand:           mean / q(p) / P(T≤x) / censored-time model; quantile definition
Dependence unit:    request, connection, process run, host, shard, time block
Estimate:           absolute values and treatment contrast
Uncertainty:        method, assumptions, interval; practical threshold
Threats:            censoring, omission, routing bias, resets, schema/version changes
Decision:           ship, reject, collect more evidence; guardrails and rollback

References

• Histograms and aggregation — HdrHistogram sizing and thread safety, Micrometer and Prometheus bucket configuration, and the correct aggregation query. Read when configuring metrics or when a dashboard's percentile is suspect.
• Comparing two measurements — tail resolution, experimental units, hierarchical replication, paired contrasts, resampling and how to report the difference. Read at step 8, whenever two p99s are about to be called different or equal.
• Coordinated omission — what it is, why it misleads in two directions at once, and how to detect and correct it. Read when the number came from a load generator or a fixed-rate producer.