# Go Perf Godebug And Metrics

> Guides Go runtime observability — the GODEBUG trace knobs (gctrace, schedtrace/scheddetail, inittrace, allocfreetrace, madvdontneed) and reading a gctrace line, the low-overhead runtime/metrics package and its histogram metrics, preferred over runtime.ReadMemStats which stops the world, and exposing metrics via expvar/Prometheus. Fires on "GODEBUG", "gctrace", "read runtime metrics", "monitor GC in production", "ReadMemStats vs runtime/metrics". Routes GC to go-perf-gc-tuning, scheduler to go-perf-goroutines-scheduler.

- Skill: `ctoth/go-perf-godebug-and-metrics` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-godebug-and-metrics`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-godebug-and-metrics/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-godebug-and-metrics

---


# Go GODEBUG & runtime/metrics

This skill owns **runtime observability for performance**: the `GODEBUG` knobs that make the runtime *narrate* what it is doing, and the `runtime/metrics` package that lets a process *read its own* runtime stats cheaply. It is a reference skill — it tells you *what to look at*; the *action* you take routes to a sibling (`go-perf-gc-tuning` for GC knobs, `go-perf-goroutines-scheduler` for scheduler depth, `go-perf-tail-latency` for the histograms, `go-perf-execution-tracer` for the tracer, `go-perf-os-tooling` for OS profilers).

All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.25 / 1.26.

---

## 1. GODEBUG — Two Jobs in One Variable

`GODEBUG` is **a comma-separated list of `key=value` settings** that "controls the execution of certain parts of a Go program" ([godebug](https://go.dev/doc/godebug)). It carries two unrelated kinds of setting, and conflating them is a common error:

- **Compatibility toggles** — Go's backwards-compatibility escape hatch. When fixing a bug would break code that "depends on buggy (including insecure) behavior," GODEBUG lets old code keep the old behavior ([godebug](https://go.dev/doc/godebug)). These have *defaults* derived from the `go` line in `go.mod`, a `godebug` directive, or a `//go:debug` line — e.g. `panicnil=1`, `http2client=0`, and the perf-relevant **`containermaxprocs`** (`{Name: "containermaxprocs", Package: "runtime", Changed: 25, Old: "0"}` — toggles the Go 1.25 container-aware `GOMAXPROCS`) ([godebugs/table.go](https://github.com/golang/go/blob/master/src/internal/godebugs/table.go)). Each compatibility setting also exports a counter `/godebug/non-default-behavior/<name>:events` so you can see in production whether old behavior is firing ([godebug](https://go.dev/doc/godebug); [metrics/description.go](https://github.com/golang/go/blob/master/src/runtime/metrics/description.go)).
- **Runtime *trace* knobs** (§2) — diagnostic output to stderr. These are **not** compatibility settings; they are debugging instrumentation set only in the environment (`GODEBUG=gctrace=1 ./server`). "Unrecognized settings in the `GODEBUG` environment variable are ignored" ([godebug](https://go.dev/doc/godebug)) — so a typo silently does nothing.

---

## 2. The Trace Knobs (set in the environment, read stderr)

Each knob below is documented in `runtime`'s package doc ([runtime: Environment Variables](https://pkg.go.dev/runtime#hdr-Environment_Variables), mirrored in `src/runtime/extern.go`). They emit lines to **standard error**; the formats are "subject to change."

| `GODEBUG=` | What it emits | Use it to see |
|---|---|---|
| `gctrace=1` | one line **per GC** (§3) | GC frequency, pause, heap sizes, GC CPU% |
| `schedtrace=N` | one line **every N ms** summarizing scheduler state | run-queue depth, idle Ps/Ms |
| `schedtrace=N,scheddetail=1` | detailed multiline per-P/M/G state every N ms | which goroutines are runnable/blocked |
| `inittrace=1` | one line **per package** with init work (§3) | slow / allocation-heavy package `init` |
| `scavtrace=1` | one line ~per GC cycle from the scavenger | memory returned to the OS, RSS behavior |
| `madvdontneed=0` | (behavior toggle) use `MADV_FREE` not `MADV_DONTNEED` on Linux | why RSS stays high until memory pressure |

Two cautions:

- **`schedtrace`/`scheddetail` and `gctrace` are diagnostics, not always-on.** `scheddetail=1` is verbose; combining several traces interleaves their lines on the same stderr and makes both unreadable. Set one at a time.
- **`allocfreetrace` is gone.** It historically emitted a stack trace on **every allocation and free** — astronomically heavy and only ever a last resort. It has been **removed** from the modern runtime (no longer present in `src/runtime/extern.go` or `src/runtime/*.go` as of the Go 1.26 tree). For per-allocation insight today, use a **heap profile** (`go-perf-pprof-profiling`) or the **execution tracer** (`go-perf-execution-tracer`), not a GODEBUG knob.

---

## 3. Reading a `gctrace` Line Field-by-Field

`gctrace=1` "causes the garbage collector to emit a single line to standard error at each collection, summarizing the amount of memory collected and the length of the pause" ([runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables)). The format (verbatim, "subject to change"):

```text
gc # @#s #%: #+#+# ms clock, #+#/#/#+# ms cpu, #->#-># MB, # MB goal, # MB stacks, #MB globals, # P
```

Field by field, **with the `runtime/metrics` metric that supersedes it** (the doc lists these mappings inline):

| Field | Meaning | Modern metric |
|---|---|---|
| `gc #` | the GC number, incremented each GC | — |
| `@#s` | seconds since program start | — |
| `#%` | **percentage of time spent in GC since program start** | `/cpu/classes/gc/total:cpu-seconds` |
| `#+#+# ms clock` | wall-clock time for the GC phases | — |
| `#+#/#/#+# ms cpu` | CPU time across the phases | `/cpu/classes/gc/*` |
| `#->#-># MB` | heap size **at GC start, at GC end, and live heap** | `/gc/scan/heap:bytes` |
| `# MB goal` | goal heap size | `/gc/heap/goal:bytes` |
| `# MB stacks` | estimated scannable stack size | `/gc/scan/stack:bytes` |
| `#MB globals` | scannable global size | `/gc/scan/globals:bytes` |
| `# P` | number of processors used | `/sched/gomaxprocs:threads` |

"The phases are stop-the-world (STW) sweep termination, concurrent mark and scan, and STW mark termination. The CPU times for mark/scan are broken down in to assist time (GC performed in line with allocation), background GC time, and idle GC time. If the line ends with `(forced)`, this GC was forced by a `runtime.GC()` call" ([runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables)). **The `#%` GC-CPU field and the `#->#->#` live-heap field are the two to read first**: a rising `#%` means GC is eating throughput, and a live heap that climbs run-to-run means a leak or a `GOMEMLIMIT` worth setting (act → `go-perf-gc-tuning`).

`inittrace=1` is read the same way — `init # @#ms, # ms clock, # bytes, # allocs` per package, surfacing a package whose `init` is slow or allocates heavily ([runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables)).

---

## 4. `runtime/metrics` — The Modern, Low-Overhead Way

For anything *programmatic* (dashboards, alerts, in-process budgets), read `runtime/metrics`, **not** `runtime.ReadMemStats`. The package "provides a stable interface to access implementation-defined metrics exported by the Go runtime … similar to existing functions like `runtime.ReadMemStats` … but significantly more general" ([runtime/metrics](https://pkg.go.dev/runtime/metrics)). Metrics are keyed by **string names with an embedded unit** (`/path:unit`), discovered at runtime via `metrics.All()`, so the set grows across Go versions without breaking your code; the `Kind` of a given metric is guaranteed stable.

```go
package main

import (
	"fmt"
	"runtime/metrics"
)

// sampleGC reads three runtime metrics in one call. Reuse the []metrics.Sample
// slice across calls; metrics.Read fills in each Value in place.
func sampleGC() {
	samples := []metrics.Sample{
		{Name: "/gc/heap/allocs:bytes"},        // cumulative bytes allocated
		{Name: "/sched/goroutines:goroutines"}, // live goroutines
		{Name: "/sched/latencies:seconds"},     // runnable→running wait (histogram)
	}
	metrics.Read(samples)

	for _, s := range samples {
		switch s.Value.Kind() {
		case metrics.KindUint64:
			fmt.Printf("%s = %d\n", s.Name, s.Value.Uint64())
		case metrics.KindFloat64:
			fmt.Printf("%s = %f\n", s.Name, s.Value.Float64())
		case metrics.KindFloat64Histogram:
			h := s.Value.Float64Histogram()
			fmt.Printf("%s = histogram with %d buckets\n", s.Name, len(h.Counts))
		case metrics.KindBad:
			panic("unsupported metric " + s.Name) // name not in metrics.All()
		}
	}
}
```

Key API ([runtime/metrics](https://pkg.go.dev/runtime/metrics)):

- **`metrics.Read([]Sample)`** fills each `Sample.Value` in place; reuse the slice between calls — the read is cheap and **does not stop the world**.
- **`metrics.All() []Description`** lists every supported metric (`Name`, `Description`, `Kind`, `Cumulative`). Iterate it rather than hardcoding names, so you skip a `KindBad` on an older runtime.
- **Histograms** (`KindFloat64Histogram`) carry `Buckets []float64` and `Counts []uint64` — the raw material for a p99 without coordinated omission (methodology → `go-perf-tail-latency`).

### A few metric names worth knowing (from `metrics/description.go`)

| Metric | Kind | Reads |
|---|---|---|
| `/gc/heap/allocs:bytes` | uint64, cumulative | total bytes ever allocated (allocation *rate* = its slope) |
| `/gc/heap/allocs:objects` | uint64, cumulative | total heap objects allocated |
| `/gc/heap/live:bytes` | uint64 | live bytes marked by the previous GC |
| `/gc/gogc:percent`, `/gc/gomemlimit:bytes` | uint64 | the effective `GOGC` / `GOMEMLIMIT` |
| `/sched/latencies:seconds` | histogram | goroutine runnable→running wait (scheduler saturation) |
| `/sched/goroutines:goroutines` | uint64 | live goroutines (leak detection) |
| `/sync/mutex/wait/total:seconds` | float64, cumulative | global lock-contention trend |
| `/memory/classes/total:bytes` | uint64 | all memory mapped read-write by the runtime |

**Deprecation to know:** `/gc/pauses:seconds` is now documented as *"Deprecated. Prefer the identical `/sched/pauses/total/gc:seconds`."* ([metrics/description.go](https://github.com/golang/go/blob/master/src/runtime/metrics/description.go)). The newer GC STW pause histograms are `/sched/pauses/total/gc:seconds` (full pause) and `/sched/pauses/stopping/gc:seconds` (just the time to stop all Ps); non-GC STW events live under `/sched/pauses/*/other:seconds`. Read code emitting `/gc/pauses:seconds` and migrate it.

---

## 5. Why `runtime/metrics` Beats `runtime.ReadMemStats`

`runtime.ReadMemStats(m *MemStats)` **stops the world** to take its snapshot — its implementation begins `stw := stopTheWorld(stwReadMemStats)` ([src/runtime/mstats.go](https://github.com/golang/go/blob/master/src/runtime/mstats.go)). Calling it on a timer in a hot service injects a global pause on every scrape, and it returns one coarse, fixed `MemStats` struct that cannot express histograms or grow with the runtime. `runtime/metrics` is the general, forward-compatible, **non-STW** replacement (§4). Prefer it for all new code; reach for `ReadMemStats` only when interfacing with old code that already depends on a specific `MemStats` field.

---

## 6. Exposing Metrics Continuously

A one-shot `metrics.Read` answers "now"; production wants a *time series*.

- **`expvar`** is the zero-dependency option. It "provides a standardized interface to public variables … exposed via HTTP at `/debug/vars` in JSON format," and importing it for its side effect registers a handler plus `cmdline` and `memstats` automatically ([expvar](https://pkg.go.dev/expvar)). Publish your own with `expvar.NewInt("requests")` or `expvar.Publish(name, v)` (`v` is any `Var` whose `String()` returns valid JSON). Bridge `runtime/metrics` into it with an `expvar.Func`:

```go
import (
	"expvar"
	"runtime/metrics"
)

func init() {
	const goroutines = "/sched/goroutines:goroutines"
	expvar.Publish("go_goroutines", expvar.Func(func() any {
		s := []metrics.Sample{{Name: goroutines}}
		metrics.Read(s)
		return s[0].Value.Uint64()
	}))
}
```

- **Prometheus** is the standard for real fleets: the official client's `collectors.NewGoCollector` already sources from `runtime/metrics`, so you get `/gc/*` and `/sched/*` on a scrape endpoint with no custom wiring. (Use it over hand-rolled gauges; correctness of the client API is out of scope here — this skill's claim is only *which* source to pull from: `runtime/metrics`, not `ReadMemStats`.)

Note `expvar`'s default `/debug/vars` handler — like `net/http/pprof` — should not be on a public port; gate it to an internal interface.

---

## 7. Routing to Related Skills

**This plugin (`go-perf-*`):**

- `go-perf-gc-tuning` — *acting* on a bad `gctrace` line or `/gc/*` metric: `GOGC`, `GOMEMLIMIT`, the pacer, allocation rate as the primary lever.
- `go-perf-goroutines-scheduler` — interpreting `schedtrace`/`/sched/*` depth: GMP model, work-stealing, container-aware `GOMAXPROCS`.
- `go-perf-tail-latency` — turning `/sched/latencies:seconds` and `/sched/pauses/total/gc:seconds` histograms into p99/p999 without coordinated omission.
- `go-perf-execution-tracer` — when you need a *timeline*, not a counter: `runtime/trace`, `go tool trace`, the flight recorder.
- `go-perf-os-tooling` — OS-level profilers (`perf`, flame graphs) when the cost is below the runtime.
- `go-perf-methodology` — the decision layer: *which* diagnostic answers *which* question.

**Parent / sibling marketplace (`go-*`):**

- `go-slog-logging` — structured logging, distinct from metric counters.
- `go-version-feature-map` — version floors: container-aware `GOMAXPROCS` (1.25), Green Tea GC default (1.26).

---

## 8. Don't

- **Don't poll `runtime.ReadMemStats` on a timer** — it stops the world every call ([mstats.go](https://github.com/golang/go/blob/master/src/runtime/mstats.go)); use `runtime/metrics` (§5).
- **Don't hardcode `/gc/pauses:seconds`** — it is deprecated in favor of the identical `/sched/pauses/total/gc:seconds` ([description.go](https://github.com/golang/go/blob/master/src/runtime/metrics/description.go)).
- **Don't reach for `allocfreetrace`** — it was removed; use a heap profile or the tracer (§2).
- **Don't leave `scheddetail=1` (or several traces) running together** — verbose, interleaved stderr that drowns the signal (§2).
- **Don't typo a GODEBUG key** — unrecognized settings are silently ignored, so the trace just never appears ([godebug](https://go.dev/doc/godebug)).
- **Don't confuse a GODEBUG *compatibility* toggle with a *trace* knob** — the first has go.mod-derived defaults and a `/godebug/non-default-behavior/*` counter; the second is environment-only stderr instrumentation (§1).
- **Don't expose `/debug/vars` (or pprof) on a public port** — gate it internally (§6).
- **Don't read `runtime/metrics` and then guess at the fix here** — act in `go-perf-gc-tuning` / `go-perf-goroutines-scheduler`; this skill only tells you *what to look at*.

---

## 9. Reference Files

Wrong/right anti-patterns in LLM-generated runtime-observability code, with citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)