Go Perf Tail Latency

Go Tail-Latency Engineering

"Reducing GC frequency may also lead to latency improvements." — A Guide to the Go Garbage Collector, Latency section

go-perf-methodology §8 introduces tail latency and names coordinated omission; this skill is its depth peer for latency-sensitive services. It owns why the tail is the metric, coordinated omission at mechanism depth, and the Go-specific tail levers in priority order. GC knobs route to go-perf-gc-tuning; capturing the trace at a spike routes to go-perf-execution-tracer; contention routes to go-perf-block-mutex-profiles. All Go below builds clean under gofmt and go vet on Go 1.26.

1. Optimize the Tail, Not the Mean

For a request-serving system, the number a user (and your SLO) feels is the tail — p99, p999, p9999 — not the average. Two reasons make the mean misleading:

• A user's session touches many requests. If one page makes 20 backend calls, it is as slow as its slowest call; a p99-per-call becomes roughly a p80-per-page — the rare slow request is not rare from the user's seat.
• Latency is one-sided and heavy-tailed. GC, scheduling, and contention only ever make a request slower, so the distribution has a long right tail the mean buries: latency "is a product of the moment-to-moment execution of the program and not just an aggregation of costs" (GC guide).

Report percentiles, and never average percentiles. The mean of two machines' p99s (or two time windows') is meaningless. Merge the underlying histograms, then read the quantile from the merge (§3).

2. Coordinated Omission — The Measurement Trap (Depth)

go-perf-methodology §8 states the headline; here is the mechanism and fix in full, because a load test with coordinated omission reports a tail off by orders of magnitude and sends you optimizing the wrong thing.

The closed-loop trap. Most load generators are closed-loop: each connection sends a request, waits for the response, then sends the next. That generator coordinates with the server — when the server stalls for 2 s the generator is also parked for 2 s, never issuing the dozens of requests a real open arrival process would have fired during the stall. Gil Tene's wrk2 names the bug:

"Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods." (wrk2 README, Gil Tene)

So one 2 s sample is recorded where there should have been that sample plus ~hundreds in [0, 2 s] behind the stall — the tail is optimistic by orders of magnitude the moment the system struggles.

The fix — open-loop + intended-start-time correction. Drive a constant request rate independent of responses, and measure each request from when it should have started, not when it went out:

"Rather than measure response latency from the time that the actual transmission of a request occurred, wrk2 measures response latency from the time the transmission should have occurred according to the constant throughput configured for the run. When responses take longer than normal (arriving later than the next request should have been sent), the true latency of the subsequent requests will be appropriately reflected." (wrk2 README)

Tools that do this correctly:

• wrk2 — wrk2 -R <rate> -t<threads> -c<conns> -d<dur> URL; the -R/--rate constant-throughput flag is the point, and it records into HdrHistograms (lossless, fixed-precision, so high quantiles are exact) for "accurate latency details to the high 9s … accurate 99.9999%'ile when run long enough" (wrk2).
• Vegeta — vegeta attack -rate=<N> is open-loop by construction (fixed rate regardless of in-flight responses) and reports percentiles.

If you must keep a closed-loop client, apply the intended-time correction yourself and never report its raw latencies as a tail.

3. Measuring Tails in Go — Histograms and runtime/metrics

Use histograms, not a running average or a min/max. Go's runtime/metrics already exposes the two latency distributions that explain most Go tails, as Float64Histogram values with unit seconds — buckets plus monotonic counts you can quantile directly:

Read a quantile straight off the histogram (this is the right shape for an SLO dashboard):

package latency

import "runtime/metrics"

// schedLatencyQuantile returns an approximate q-quantile (e.g. 0.99) of the
// runnable→running scheduler latency, in seconds. A rising p99 here is an
// off-CPU stall — goroutines are ready but not getting a P — which a CPU
// profile cannot see (route diagnosis to go-perf-execution-tracer).
func schedLatencyQuantile(q float64) float64 {
	s := []metrics.Sample{{Name: "/sched/latencies:seconds"}}
	metrics.Read(s)
	return histQuantile(s[0].Value.Float64Histogram(), q)
}

func histQuantile(h *metrics.Float64Histogram, q float64) float64 {
	var total uint64
	for _, c := range h.Counts {
		total += c
	}
	if total == 0 {
		return 0
	}
	target := uint64(float64(total) * q)
	var cum uint64
	for i, c := range h.Counts {
		if cum += c; cum >= target {
			return h.Buckets[i+1] // upper edge of the containing bucket
		}
	}
	return h.Buckets[len(h.Buckets)-1]
}

These histograms have monotonically increasing bucket counts (cumulative since start), so for a live p99 you diff two reads over your window before quantiling. The full runtime/metrics catalog and GODEBUG traces route to go-perf-godebug-and-metrics.

Catch the spike, don't average it away. A p999 event is by definition rare; a periodic sampler or a CPU profile averaged over a minute washes it out. Use the execution tracer's flight recorder (trace.FlightRecorder, Go 1.25): keep a sliding in-memory trace window and snapshot it after a slow request fires, so you capture the trace of the incident, not the calm after. Mechanism → go-perf-execution-tracer.

4. The Go Tail Levers, In Priority Order

Spend effort top-down. In Go the dominant tail cost is usually the GC's ambient tax on running goroutines, not its stop-the-world pause.

Lever 1 — Lower the allocation rate (the dominant lever)

Fewer bytes/sec allocated → fewer GC cycles → less of every GC-induced latency below: "reducing GC frequency may also lead to latency improvements … not only [from] tuning parameters, like increasing GOGC and/or the memory limit, but also … the optimizations [that cut allocation]" (GC guide). What makes this the tail lever, not just a throughput one, is mark assist: when a goroutine allocates while a GC is running, the runtime makes that goroutine pay down GC debt first — runtime.gcAssistAlloc is where "goroutines … yield some of their time to assist the GC with scanning and marking," and ">5%" cumulative time there "indicates that the application is likely out-pacing the GC with respect to how fast it's allocating" (GC guide; a listed latency source is "User goroutines assisting the GC in response to a high allocation rate"). A request that allocates during a GC eats the assist tax — that is your bimodal tail. Cut allocations at the source → go-perf-allocator-internals, go-perf-sync-pool, go-perf-escape-analysis.

Lever 2 — Give the GC headroom (knobs, second)

Once allocation is as low as practical, raise the heap target so cycles are less frequent: a higher GOGC and/or a GOMEMLIMIT set with headroom below the real cap. This trades memory for fewer GCs and thus fewer assist windows. It is lever 2, not 1 — turning the knob without cutting allocation just moves the same tax around. All GOGC/GOMEMLIMIT/pacer/Green-Tea detail → go-perf-gc-tuning.

Lever 3 — Scheduler latency (runnable but not running)

A goroutine can be ready to run yet wait for a P. That wait is invisible to a CPU profile and shows up in /sched/latencies:seconds (§3). Causes: GOMAXPROCS too low for the load (or a stale container-aware value), too many runnable goroutines, or long GC mark phases that take "25% of CPU resources when in the mark phase" (GC guide) away from your handlers. Diagnose with the tracer's scheduler view → go-perf-execution-tracer; scheduler/GOMAXPROCS depth → go-perf-goroutines-scheduler.

Lever 4 — Lock-contention spikes

A sync.Mutex that is cheap on average can spike the tail when many goroutines pile up on it during a burst. /sync/mutex/wait/total:seconds flags a global change; the block/mutex profiles name the exact lock — they record nothing until enabled (runtime.SetMutexProfileFraction, runtime.SetBlockProfileRate). Diagnosis → go-perf-block-mutex-profiles; the fix (sharding, batching) → go-perf-contention-and-sharding.

Lever 5 — Huge single allocations and large pointer-ful heaps

A multi-megabyte allocation, or a large map/slice full of pointers, lengthens the mark phase — the collector must scan every pointer ("Pointer writes requiring additional work while the GC is in the mark phase" and "Running goroutines must be suspended for their roots to be scanned" are both listed latency sources (GC guide)). Pointer-free (no-scan) data is skipped by the scanner — prefer value layouts and indices-into-a-slice over pointer graphs on hot, large structures. Internals → go-perf-allocator-internals; layout → go-perf-struct-layout.

Note on STW: Go's collector "is not fully stop-the-world and does most of its work concurrently … primarily to reduce application latencies" (GC guide). STW pauses are brief transitions; for most services the assist tax (lever 1) and scheduling (lever 3) dominate the tail, not the pause — do not start at "tune the GC pause."

5. Don't

• Don't report the mean/median for a latency SLO — report p99/p999 (GC guide: latency is not an aggregation of costs).
• Don't trust a closed-loop load generator's tail — it coordinate-omits the worst latencies; use an open-loop, constant-rate tool with intended-start-time correction (wrk2).
• Don't average percentiles across machines or windows — merge the histograms, then quantile.
• Don't reach for GOGC/GOMEMLIMIT first — the dominant lever is fewer allocations → fewer GCs → less mark-assist tax; the knob is lever 2 (GC guide).
• Don't blame the STW pause for the tail — Go STW is a brief transition; the assist tax and scheduler latency usually dominate (GC guide).
• Don't ignore /sched/latencies:seconds — a runnable-but-not-running stall is off-CPU and a CPU profile is blind to it (runtime/metrics).
• Don't use the deprecated /gc/pauses:seconds — prefer /sched/pauses/total/gc:seconds (runtime/metrics).
• Don't catch a p999 spike with a periodic sampler — use the flight recorder to snapshot the trace at the spike → go-perf-execution-tracer.

6. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-gc-tuning — GOGC/GOMEMLIMIT, pacer, Green Tea GC, gctrace (lever 2).
• go-perf-execution-tracer — runtime/trace, go tool trace, the flight recorder to capture a trace at a latency spike; scheduler-latency view.
• go-perf-block-mutex-profiles — block/mutex/goroutine profiles for contention spikes (lever 4).
• go-perf-allocator-internals, go-perf-sync-pool, go-perf-escape-analysis — cutting allocation rate at the source (lever 1).
• go-perf-goroutines-scheduler — GOMAXPROCS, GMP, container-aware defaults (lever 3); go-perf-struct-layout — pointer-free layouts that shorten GC scans (lever 5).
• go-perf-godebug-and-metrics — the full runtime/metrics catalog; go-perf-methodology §8 — the tail-latency/coordinated-omission intro this skill deepens.

Parent / sibling marketplace (go-* correctness): go-performance (measure-first overview); go-sync-primitives, go-race-and-memory-model (lock and memory-ordering correctness).

7. Reference Files

Tail-latency anti-patterns in LLM-generated Go, each with a wrong/right contrast and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml