# Go Perf Production Incidents

> Guides diagnosing a LIVE Go performance incident under on-call pressure — OOMKill / RSS climb, GC death-spiral / CPU pegged in GC, p99 latency cliff, goroutine leak / pile-up, CPU saturation — with the capture-evidence-before-restart discipline, safe prod pprof / flight-recorder / heap-profile runbooks, and GOMEMLIMIT triage. Auto-invokes on "my service is OOMing", "production latency spiked", "pod keeps getting OOMKilled", "diagnose this incident", "goroutines climbing in prod", "CPU pegged in GC". Routes GC knobs to go-perf-gc-tuning, tail latency to go-perf-tail-latency, the tracer/flight recorder to go-perf-execution-tracer.

- Skill: `ctoth/go-perf-production-incidents` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-production-incidents`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-production-incidents/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-production-incidents

---


# Go Performance Production Incidents

> "Your first response in a major outage may be to start troubleshooting and try to find a root cause as quickly as possible. Ignore that instinct!"
> — Google SRE, [Effective Troubleshooting](https://sre.google/sre-book/effective-troubleshooting/)

This is the **human / on-call layer** of Go performance: a service is on fire *right now*, you are tired, it is 3am, and the wrong move makes the diagnosis impossible. The other `go-perf-*` skills own the theory (which knob, which profiler, why one run lies). This one owns the **lived incident** — the five failures you actually get paged for and the order to handle them in. The single rule that survives the panic: **capture evidence before you restart.** A restart "fixes" the symptom and *destroys* the heap profile, the goroutine dump, and the trace that tell you why — so the incident recurs next week with no new data.

All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.25/1.26.

**Triage index — match the symptom, jump to the runbook:**

| What you're seeing | First evidence to capture | Section |
|---|---|---|
| Pod OOMKilled (exit 137), RSS climbing | heap profile (`inuse_space`) + goroutine profile | §3 |
| CPU 100%, throughput dead, `gctrace` GC% high & rising | `GODEBUG=gctrace=1` line | §4 |
| p99/p999 cliff, mean fine | flight-recorder snapshot at the spike | §5 |
| Goroutine count climbing without bound | goroutine profile (`debug=1`) | §6 |
| CPU pegged, `gctrace` GC% *low* | 30s prod CPU profile | §7 |

---

## 1. The calm order — stabilize, capture, diagnose, fix, gate

Under pressure the instinct is to jump straight to root cause. SRE doctrine is the opposite: "Stopping the bleeding should be your first priority; you aren't helping your users if the system dies while you're root-causing" ([Effective Troubleshooting](https://sre.google/sre-book/effective-troubleshooting/)). The fixed order:

1. **Stabilize** — roll back the bad deploy, scale out, shed load, or drain the sick pod. Restore service first: "Stop the bleeding, restore service, and **preserve the evidence for root-causing**" ([Managing Incidents](https://sre.google/sre-book/managing-incidents/)).
2. **Capture evidence — *before* the restart.** Pull the heap profile, the goroutine profile, the trace snapshot off the *sick* instance while it is still sick. This is the step everyone skips and the one that decides whether you ever find the bug. If you must kill the pod, grab profiles from it first, or keep **one** sick replica cordoned for forensics.
3. **Diagnose** from the captured artifacts, calmly, after service is restored — not in the middle of the page.
4. **Fix** the root cause (an allocation, a leak, a missing `GOMEMLIMIT`, an unbounded goroutine spawn).
5. **Add a regression gate** so it can't silently return — an alloc/goroutine budget or a benchmark in CI (→ `go-perf-methodology` §9).

Resist the classic traps while diagnosing: "latching on to causes of past problems, reasoning that since it happened once, it must be happening again," and remember "correlation is not causation" ([Effective Troubleshooting](https://sre.google/sre-book/effective-troubleshooting/)). Capture first, theorize later.

---

## 2. Safe production profiling — the discipline that keeps profiling from becoming the incident

`net/http/pprof` registers handlers under `/debug/pprof/`; you "typically only import [it] for the side effect of registering its HTTP handlers" ([net/http/pprof](https://pkg.go.dev/net/http/pprof)). Three rules make it safe to use on a live, paying system:

- **Never on the public port.** Mount pprof on a **private admin listener** bound to localhost or the internal network, never the same mux that serves user traffic — `/debug/pprof/` is an unauthenticated heap/goroutine/CPU dump of your process and a denial-of-service handle. (Experiential judgment; the safe-exposure mechanics live in **`go-perf-pprof-profiling`**.)

  ```go
  // Separate admin mux on a private port — NOT http.DefaultServeMux on :443.
  adminMux := http.NewServeMux()
  adminMux.HandleFunc("/debug/pprof/", pprof.Index)
  adminMux.HandleFunc("/debug/pprof/profile", pprof.Profile)
  go func() { log.Fatal(http.ListenAndServe("127.0.0.1:6060", adminMux)) }()
  ```

- **One profile at a time.** A CPU profile perturbs the very latency you're measuring, and "precise memory profiling skews CPU profiles" ([Diagnostics](https://go.dev/doc/diagnostics)). Take the CPU profile, *then* the heap profile — never both at once, or both lie.
- **Know the cost.** A CPU profile (`/debug/pprof/profile?seconds=30`) costs ~30s of sampling overhead; a heap or goroutine profile is a cheap point-in-time snapshot you can take immediately. On a process with hundreds of thousands of goroutines, the goroutine dump itself briefly stops the world — expect a small hiccup, and prefer `debug=1` over the giant `debug=2` stack dump.

Fetch directly with the tool you already have ([net/http/pprof](https://pkg.go.dev/net/http/pprof)):

```bash
go tool pprof http://127.0.0.1:6060/debug/pprof/heap            # inuse_space (live) by default
go tool pprof http://127.0.0.1:6060/debug/pprof/goroutine       # leak hunt
go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30   # CPU
curl -o sick.heap http://127.0.0.1:6060/debug/pprof/heap        # save the artifact off the dying pod
```

---

## 3. OOMKill / memory blow-up — the pod keeps getting killed

**What it looks like:** the container is OOMKilled (exit 137), RSS climbing on the dashboard. The dashboard shape is the first diagnostic: a **sawtooth** RSS (climbs, GC reclaims, climbs) that crests the limit is a *transient-spike or limit-too-low* problem; a **monotonic climb** that never comes back down is a **leak** — your code is holding references the GC can't free.

Runbook:

1. **Capture the heap profile from the sick pod** — `inuse_space` is the default and shows *live* objects: it "tracks ... the allocation sites for all live objects" and pprof defaults to "`-inuse_space` (live objects, scaled by size)" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)). The top frames are who is holding the memory.
2. **Check for a goroutine leak in the same breath** (§6) — leaked goroutines hold their entire stack and captured variables live, and present *as* a memory leak. A monotonic climb is a goroutine leak until proven otherwise.
3. **Confirm `GOMEMLIMIT` is set.** With no soft limit, the runtime GCs on `GOGC` growth ratio alone and will happily let the heap sail past the container limit into an OOMKill instead of **GCing harder** as it approaches the ceiling. Setting `GOMEMLIMIT` to ~90% of the container limit makes the GC fight to stay under it — "leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of" ([GC Guide](https://go.dev/doc/gc-guide)). Knob details → **`go-perf-gc-tuning`**.

The trap to refuse: **do not "just bump the memory limit"** on a monotonic climb. A leak grows without bound; a bigger limit only buys a longer fuse before the same OOMKill, and you've thrown away the heap profile that names the leak. Bump the limit only to confirm the shape is a *sawtooth* (real working set) and not a leak.

---

## 4. GC death-spiral — CPU pegged in GC

**What it looks like:** CPU at 100% but throughput collapsed, p99 exploded, and `GODEBUG=gctrace=1` shows GCs firing back-to-back with the GC-time percentage high and rising:

```text
gc 412 @30.1s 58%: 0.10+820+0.05 ms clock, ... 980->998->995 MB, 1000 MB goal, 8 P
gc 413 @30.9s 59%: 0.09+810+0.06 ms clock, ... 996->999->996 MB, 1000 MB goal, 8 P
```

The `58%`/`59%` is the fraction of total time spent in GC since start (→ `go-perf-gc-tuning` §6 reads this line), and a live heap (`996 MB`) pinned against the goal (`1000 MB`) is the signature. This is **thrashing**: "the program fails to make reasonable progress due to constant GC cycles ... It's particularly dangerous because it effectively stalls the program" ([GC Guide](https://go.dev/doc/gc-guide)). The usual cause is a **`GOMEMLIMIT` set too tight** (or `GOGC` too low) for the live working set — the GC is being asked to maintain "an impossible memory limit."

The decision under pressure: **relax the limit.** A too-tight `GOMEMLIMIT` trades an OOM for an indefinite stall, and "in many cases, an indefinite stall is worse than an out-of-memory condition, which tends to result in a much faster failure" ([GC Guide](https://go.dev/doc/gc-guide)). Raise the limit (or `GOGC`) to stop the spiral now; the *durable* fix is cutting the allocation rate so the live heap sits well below the ceiling. Which knob, by how much → **`go-perf-gc-tuning`**.

---

## 5. p99 latency cliff — capture WITH the flight recorder, at the spike

**What it looks like:** mean latency fine, p99/p999 suddenly cliff-edges. A CPU profile is the *wrong* tool — it is "blind to off-CPU time," and a request slow because it's blocked on a lock, a channel, GC mark-assist, or the scheduler contributes nothing to it ([Diagnostics](https://go.dev/doc/diagnostics)). The right tool is the **Go 1.25 flight recorder**, which exists precisely because by the time you notice a rare spike "it's already too late to call `Start`" ([Flight Recorder](https://go.dev/blog/flight-recorder)).

It keeps the last few seconds of execution trace in a ring buffer in memory and lets you **snapshot the window *after* the spike has already happened** — "the power of hindsight ... capture just what went wrong, after it's already happened" ([Flight Recorder](https://go.dev/blog/flight-recorder)):

```go
fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{
    MinAge:   200 * time.Millisecond, // ~2x the event window you care about
    MaxBytes: 1 << 20,                // bound the memory cost
})
fr.Start()
// ... in the request path, snapshot only when this request was pathologically slow:
if fr.Enabled() && time.Since(start) > 100*time.Millisecond {
    go captureSnapshot(fr) // fr.WriteTo(f) — only the first call has effect
}
```

Then open the snapshot in `go tool trace` and read scheduler-latency and GC mark-assist at the spike (tracer mechanics → **`go-perf-execution-tracer`**; tail-latency method, coordinated omission, the alloc-rate lever → **`go-perf-tail-latency`**). The dominant tail lever is usually **lowering the allocation rate**, not a GC knob — Go STW pauses are sub-millisecond and rarely dominate the tail.

---

## 6. Goroutine leak / pile-up — the count climbs without bound

**What it looks like:** goroutine count rising monotonically on the dashboard; memory and scheduler pressure follow. The diagnostic is the **goroutine profile** — "stack traces of all current goroutines" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)) — and the tell is unmistakable: thousands of goroutines stuck on the **same line**, blocked on a channel send/receive or a mutex that will never be released.

```bash
# Top stacks, aggregated — the leak is the one with a count in the thousands and climbing.
go tool pprof -top http://127.0.0.1:6060/debug/pprof/goroutine
# Or the readable full dump for the exact blocked line (debug=1, not the huge debug=2):
curl 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' -o goroutines.txt
```

A leak count that grows across two snapshots 30s apart confirms it. This is the same symptom that masquerades as an OOM (§3): each leaked goroutine pins its stack and captured variables. The *fix* is correctness — a missing `context` cancellation, an unbounded spawn, a send on a channel with no receiver. Goroutine-lifecycle and leak-prevention discipline → **`go-concurrency-goroutines`**; block/mutex profiling to find *what* they're stuck on → **`go-perf-block-mutex-profiles`**.

---

## 7. CPU saturation / runaway loop

**What it looks like:** CPU pegged but — unlike §4 — `gctrace` shows a *low* GC percentage, so it's mutator work, not the collector. This is the one case the CPU profile is exactly right for: it "determines where a program spends its time while actively consuming CPU cycles" ([Diagnostics](https://go.dev/doc/diagnostics)). Take a 30s production CPU profile off the admin port (§2) and read the hot frames:

```bash
go tool pprof -top http://127.0.0.1:6060/debug/pprof/profile?seconds=30
```

A single function dominating flat time is a hot path or a runaway/spin loop (a busy-wait, a regex backtracking, an O(n²) on grown input). Profile-reading depth — flat vs cumulative, `list`, sampling caveats → **`go-perf-pprof-profiling`** and **`go-perf-methodology`** §2. Before optimizing the hot function, confirm it's the bottleneck and not a symptom of saturation elsewhere (run-queue depth, lock contention) via the USE method → **`go-perf-methodology`** §3.

---

## 8. Don't

- **Don't restart before capturing evidence.** The restart resolves the symptom and erases the heap profile, goroutine dump, and trace — guaranteeing a recurrence with zero new data. Capture from the sick instance first ([Managing Incidents](https://sre.google/sre-book/managing-incidents/)).
- **Don't "just bump the memory limit" on an OOM** without checking the dashboard shape — a monotonic climb is a leak, and a bigger limit only delays the same OOMKill while you discard the heap profile that names it (§3).
- **Don't run a service with no `GOMEMLIMIT` in a fixed-memory container** — without it the runtime OOMs instead of GCing harder as it nears the ceiling; set it ~5–10% under the container limit ([GC Guide](https://go.dev/doc/gc-guide)).
- **Don't set a `GOMEMLIMIT` you can't honor.** A too-tight limit thrashes, and "an indefinite stall is worse than an out-of-memory condition" ([GC Guide](https://go.dev/doc/gc-guide)) (§4).
- **Don't reach for a CPU profile to explain a latency cliff.** It's blind to off-CPU waits (locks, channels, mark-assist); use the flight recorder + tracer ([Diagnostics](https://go.dev/doc/diagnostics)) (§5).
- **Don't run multiple profiles at once on a sick box** — "precise memory profiling skews CPU profiles" ([Diagnostics](https://go.dev/doc/diagnostics)); take them one at a time (§2).
- **Don't expose `/debug/pprof/` on the public port.** It's an unauthenticated process dump and a DoS handle — private admin listener only (§2).
- **Don't jump to a root-cause theory at 3am.** Stabilize, capture, then diagnose calmly — "Ignore that instinct!" ([Effective Troubleshooting](https://sre.google/sre-book/effective-troubleshooting/)) (§1).
- **Don't close the incident without a regression gate** — an alloc/goroutine budget or CI benchmark, or it returns silently (§1, → `go-perf-methodology` §9).

---

## 9. Routing to Related Skills

**This plugin (`go-perf-*` depth):**

- `go-perf-gc-tuning` — `GOGC`/`GOMEMLIMIT` knobs, the thrashing math, reading `gctrace`, allocation rate as the durable fix (§3, §4).
- `go-perf-tail-latency` — p99/p999 method, coordinated omission, the alloc-rate tail lever (§5).
- `go-perf-execution-tracer` — `runtime/trace`, `go tool trace`, flight-recorder mechanics (§5).
- `go-perf-pprof-profiling` — CPU/heap pprof subcommands, `inuse` vs `alloc`, safe production-profiling exposure depth (§2, §7).
- `go-perf-block-mutex-profiles` — block/mutex/goroutine profiles and the enablement gotcha (§6).
- `go-perf-godebug-and-metrics` — `runtime/metrics` catalog and `GODEBUG` for dashboards/alerts that page you in the first place.
- `go-perf-methodology` — the measure-first discipline, the tool-selection matrix, USE method, CI regression gating (§1, §7).

**Parent / sibling marketplace (`go-*` correctness):**

- `go-concurrency-goroutines` — goroutine lifecycle, cancellation, and leak *prevention* — the fix behind §6.
- `go-context` — cancellation/propagation, the usual missing piece in an unbounded-goroutine leak.

---

## 10. Reference Files

Incident anti-patterns in LLM-generated Go on-call work, each with wrong/right contrast and citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)