# Go Perf Tail Latency

> Guides tail-latency for Go services — optimizing p99/p999 not the mean; coordinated omission (Gil Tene) and open-loop/constant-rate fix (wrk2, Vegeta); the Go tail levers in order — lower allocation rate → fewer GCs → less mark-assist tax, GC headroom, scheduler latency, contention, large GC-scan allocs; histograms + flight recorder. Fires on "reduce p99", "tail latency spikes", "GC pause latency", "my latency is bimodal", "measure latency correctly". Routes GC knobs to go-perf-gc-tuning, tracer to go-perf-execution-tracer.

- Skill: `ctoth/go-perf-tail-latency` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-tail-latency`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-tail-latency/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-tail-latency

---


# Go Tail-Latency Engineering

> "Reducing GC frequency may also lead to latency improvements."
> — [A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide), Latency section

`go-perf-methodology` §8 introduces tail latency and names coordinated omission; this skill is its
**depth peer** for latency-sensitive services. It owns *why the tail is the metric*, coordinated
omission **at mechanism depth**, and the **Go-specific tail levers in priority order**. GC knobs route
to `go-perf-gc-tuning`; capturing the trace *at* a spike routes to `go-perf-execution-tracer`;
contention routes to `go-perf-block-mutex-profiles`. All Go below builds clean under `gofmt` and
`go vet` on Go 1.26.

---

## 1. Optimize the Tail, Not the Mean

For a request-serving system, the number a user (and your SLO) feels is **the tail** — p99, p999,
p9999 — not the average. Two reasons make the mean misleading:

- **A user's session touches many requests.** If one page makes 20 backend calls, it is as slow as its
  slowest call; a p99-per-call becomes roughly a p80-per-page — the rare slow request is not rare from
  the user's seat.
- **Latency is one-sided and heavy-tailed.** GC, scheduling, and contention only ever make a request
  *slower*, so the distribution has a long right tail the mean buries: latency "is a product of the
  moment-to-moment execution of the program and not just an aggregation of costs" ([GC guide](https://go.dev/doc/gc-guide)).

**Report percentiles, and never average percentiles.** The mean of two machines' p99s (or two time
windows') is meaningless. Merge the underlying **histograms**, then read the quantile from the merge (§3).

---

## 2. Coordinated Omission — The Measurement Trap (Depth)

`go-perf-methodology` §8 states the headline; here is the mechanism and fix in full, because **a load
test with coordinated omission reports a tail off by orders of magnitude** and sends you optimizing the
wrong thing.

**The closed-loop trap.** Most load generators are *closed-loop*: each connection sends a request,
waits for the response, then sends the next. That generator **coordinates with the server** — when the
server stalls for 2 s the generator is *also* parked for 2 s, never issuing the dozens of requests a
real open arrival process would have fired during the stall. Gil Tene's wrk2 names the bug:

> "Since each connection will only begin to send a request after receiving a response, high latency
> responses result in the load generator coordinating with the server to avoid measurement during high
> latency periods." ([wrk2 README](https://github.com/giltene/wrk2), Gil Tene)

So one 2 s sample is recorded where there should have been that sample **plus** ~hundreds in [0, 2 s]
behind the stall — the tail is optimistic by orders of magnitude the moment the system struggles.

**The fix — open-loop + intended-start-time correction.** Drive a **constant request rate** independent
of responses, and measure each request from when it *should* have started, not when it went out:

> "Rather than measure response latency from the time that the actual transmission of a request
> occurred, wrk2 measures response latency from the time the transmission *should* have occurred
> according to the constant throughput configured for the run. When responses take longer than normal
> (arriving later than the next request should have been sent), the true latency of the subsequent
> requests will be appropriately reflected." ([wrk2 README](https://github.com/giltene/wrk2))

**Tools that do this correctly:**

- **wrk2** — `wrk2 -R <rate> -t<threads> -c<conns> -d<dur> URL`; the `-R`/`--rate` constant-throughput
  flag is the point, and it records into **HdrHistograms** (lossless, fixed-precision, so high
  quantiles are exact) for "accurate latency details to the high 9s … accurate 99.9999%'ile when run
  long enough" ([wrk2](https://github.com/giltene/wrk2)).
- **Vegeta** — `vegeta attack -rate=<N>` is open-loop by construction (fixed rate regardless of
  in-flight responses) and reports percentiles.

If you must keep a closed-loop client, apply the intended-time correction yourself and **never** report
its raw latencies as a tail.

---

## 3. Measuring Tails in Go — Histograms and `runtime/metrics`

**Use histograms, not a running average or a min/max.** Go's `runtime/metrics` already exposes the
two latency distributions that explain most Go tails, as `Float64Histogram` values with unit
`seconds` — buckets plus monotonic counts you can quantile directly:

| Metric | What it tells you |
|---|---|
| `/sched/latencies:seconds` | "Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running" ([runtime/metrics](https://pkg.go.dev/runtime/metrics)) — runnable-but-not-running = off-CPU stall. |
| `/sched/pauses/total/gc:seconds` | GC stop-the-world pause distribution. **`/gc/pauses:seconds` is deprecated** — "Prefer the identical /sched/pauses/total/gc:seconds" ([runtime/metrics](https://pkg.go.dev/runtime/metrics)). |
| `/sync/mutex/wait/total:seconds` | "Approximate cumulative time goroutines have spent blocked on a sync.Mutex, sync.RWMutex, or runtime-internal lock … useful for identifying global changes in lock contention" ([runtime/metrics](https://pkg.go.dev/runtime/metrics)). |

Read a quantile straight off the histogram (this is the right shape for an SLO dashboard):

```go
package latency

import "runtime/metrics"

// schedLatencyQuantile returns an approximate q-quantile (e.g. 0.99) of the
// runnable→running scheduler latency, in seconds. A rising p99 here is an
// off-CPU stall — goroutines are ready but not getting a P — which a CPU
// profile cannot see (route diagnosis to go-perf-execution-tracer).
func schedLatencyQuantile(q float64) float64 {
	s := []metrics.Sample{{Name: "/sched/latencies:seconds"}}
	metrics.Read(s)
	return histQuantile(s[0].Value.Float64Histogram(), q)
}

func histQuantile(h *metrics.Float64Histogram, q float64) float64 {
	var total uint64
	for _, c := range h.Counts {
		total += c
	}
	if total == 0 {
		return 0
	}
	target := uint64(float64(total) * q)
	var cum uint64
	for i, c := range h.Counts {
		if cum += c; cum >= target {
			return h.Buckets[i+1] // upper edge of the containing bucket
		}
	}
	return h.Buckets[len(h.Buckets)-1]
}
```

These histograms have **monotonically increasing** bucket counts (cumulative since start), so for a
live p99 you diff two reads over your window before quantiling. The full `runtime/metrics` catalog and
`GODEBUG` traces route to **`go-perf-godebug-and-metrics`**.

**Catch the spike, don't average it away.** A p999 event is by definition rare; a periodic sampler or
a CPU profile averaged over a minute washes it out. Use the **execution tracer's flight recorder**
(`trace.FlightRecorder`, Go 1.25): keep a sliding in-memory trace window and snapshot it *after* a
slow request fires, so you capture the trace **of the incident**, not the calm after. Mechanism →
**`go-perf-execution-tracer`**.

---

## 4. The Go Tail Levers, In Priority Order

Spend effort top-down. In Go the dominant tail cost is usually the GC's *ambient tax on running
goroutines*, not its stop-the-world pause.

### Lever 1 — Lower the allocation rate (the dominant lever)

Fewer bytes/sec allocated → fewer GC cycles → less of every GC-induced latency below: "reducing GC
frequency may also lead to latency improvements … not only [from] tuning parameters, like increasing
GOGC and/or the memory limit, but also … the optimizations [that cut allocation]"
([GC guide](https://go.dev/doc/gc-guide)). What makes this the *tail* lever, not just a throughput one,
is **mark assist**: when a goroutine allocates while a GC is running, the runtime makes *that goroutine*
pay down GC debt first — `runtime.gcAssistAlloc` is where "goroutines … yield some of their time to
assist the GC with scanning and marking," and ">5%" cumulative time there "indicates that the
application is likely out-pacing the GC with respect to how fast it's allocating" ([GC guide](https://go.dev/doc/gc-guide);
a listed latency source is "User goroutines assisting the GC in response to a high allocation rate").
**A request that allocates during a GC eats the assist tax — that is your bimodal tail.** Cut
allocations at the source → **`go-perf-allocator-internals`**, `go-perf-sync-pool`, `go-perf-escape-analysis`.

### Lever 2 — Give the GC headroom (knobs, second)

Once allocation is as low as practical, raise the heap target so cycles are **less frequent**: a
higher `GOGC` and/or a `GOMEMLIMIT` set with headroom below the real cap. This trades memory for fewer
GCs and thus fewer assist windows. It is lever **2**, not 1 — turning the knob without cutting
allocation just moves the same tax around. All `GOGC`/`GOMEMLIMIT`/pacer/Green-Tea detail →
**`go-perf-gc-tuning`**.

### Lever 3 — Scheduler latency (runnable but not running)

A goroutine can be ready to run yet wait for a P. That wait is invisible to a CPU profile and shows up
in `/sched/latencies:seconds` (§3). Causes: `GOMAXPROCS` too low for the load (or a stale
container-aware value), too many runnable goroutines, or long GC mark phases that take "25% of CPU
resources when in the mark phase" ([GC guide](https://go.dev/doc/gc-guide)) away from your handlers.
Diagnose with the tracer's scheduler view → `go-perf-execution-tracer`; scheduler/`GOMAXPROCS` depth →
`go-perf-goroutines-scheduler`.

### Lever 4 — Lock-contention spikes

A `sync.Mutex` that is cheap on average can spike the tail when many goroutines pile up on it during a
burst. `/sync/mutex/wait/total:seconds` flags a global change; the **block/mutex profiles** name the
exact lock — they record nothing until enabled (`runtime.SetMutexProfileFraction`,
`runtime.SetBlockProfileRate`). Diagnosis → **`go-perf-block-mutex-profiles`**; the fix (sharding,
batching) → `go-perf-contention-and-sharding`.

### Lever 5 — Huge single allocations and large pointer-ful heaps

A multi-megabyte allocation, or a large map/slice **full of pointers**, lengthens the mark phase —
the collector must scan every pointer ("Pointer writes requiring additional work while the GC is in the
mark phase" and "Running goroutines must be suspended for their roots to be scanned" are both listed
latency sources ([GC guide](https://go.dev/doc/gc-guide))). Pointer-free (no-scan) data is skipped by
the scanner — prefer value layouts and indices-into-a-slice over pointer graphs on hot, large
structures. Internals → `go-perf-allocator-internals`; layout → `go-perf-struct-layout`.

**Note on STW:** Go's collector "is not fully stop-the-world and does most of its work concurrently …
primarily to reduce application latencies" ([GC guide](https://go.dev/doc/gc-guide)). STW pauses are
brief transitions; for most services the assist tax (lever 1) and scheduling (lever 3) dominate the
tail, **not** the pause — do not start at "tune the GC pause."

---

## 5. Don't

- **Don't report the mean/median for a latency SLO** — report p99/p999 ([GC guide](https://go.dev/doc/gc-guide): latency is not an aggregation of costs).
- **Don't trust a closed-loop load generator's tail** — it coordinate-omits the worst latencies; use an open-loop, constant-rate tool with intended-start-time correction ([wrk2](https://github.com/giltene/wrk2)).
- **Don't average percentiles** across machines or windows — merge the histograms, then quantile.
- **Don't reach for `GOGC`/`GOMEMLIMIT` first** — the dominant lever is fewer allocations → fewer GCs → less mark-assist tax; the knob is lever 2 ([GC guide](https://go.dev/doc/gc-guide)).
- **Don't blame the STW pause for the tail** — Go STW is a brief transition; the assist tax and scheduler latency usually dominate ([GC guide](https://go.dev/doc/gc-guide)).
- **Don't ignore `/sched/latencies:seconds`** — a runnable-but-not-running stall is off-CPU and a CPU profile is blind to it ([runtime/metrics](https://pkg.go.dev/runtime/metrics)).
- **Don't use the deprecated `/gc/pauses:seconds`** — prefer `/sched/pauses/total/gc:seconds` ([runtime/metrics](https://pkg.go.dev/runtime/metrics)).
- **Don't catch a p999 spike with a periodic sampler** — use the flight recorder to snapshot the trace at the spike → `go-perf-execution-tracer`.

---

## 6. Routing to Related Skills

**This plugin (`go-perf-*` depth):**

- `go-perf-gc-tuning` — `GOGC`/`GOMEMLIMIT`, pacer, Green Tea GC, `gctrace` (lever 2).
- `go-perf-execution-tracer` — `runtime/trace`, `go tool trace`, the **flight recorder** to capture a trace at a latency spike; scheduler-latency view.
- `go-perf-block-mutex-profiles` — block/mutex/goroutine profiles for contention spikes (lever 4).
- `go-perf-allocator-internals`, `go-perf-sync-pool`, `go-perf-escape-analysis` — cutting allocation rate at the source (lever 1).
- `go-perf-goroutines-scheduler` — `GOMAXPROCS`, GMP, container-aware defaults (lever 3); `go-perf-struct-layout` — pointer-free layouts that shorten GC scans (lever 5).
- `go-perf-godebug-and-metrics` — the full `runtime/metrics` catalog; `go-perf-methodology` §8 — the tail-latency/coordinated-omission intro this skill deepens.

**Parent / sibling marketplace (`go-*` correctness):** `go-performance` (measure-first overview);
`go-sync-primitives`, `go-race-and-memory-model` (lock and memory-ordering *correctness*).

---

## 7. Reference Files

Tail-latency anti-patterns in LLM-generated Go, each with a wrong/right contrast and citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)