# Go Perf Goroutines Scheduler

> Guides goroutine and scheduler performance — stack cost (~2KB; cheap, but unbounded spawning isn't), the G-M-P model and work-stealing, GOMAXPROCS as the parallelism limit, container-aware GOMAXPROCS (Go 1.25; cgroup CPU limit, GODEBUG containermaxprocs/updatemaxprocs, SetDefaultGOMAXPROCS), schedtrace, syscall/M handoff. Fires on "how many goroutines is too many", "set GOMAXPROCS", "goroutines not parallel", "container CPU limit Go". Routes leaks to go-concurrency-goroutines, pool sizing to go-perf-worker-pools-throughput.

- Skill: `ctoth/go-perf-goroutines-scheduler` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-goroutines-scheduler`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-goroutines-scheduler/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-goroutines-scheduler

---


# Go Performance: Goroutines & the Scheduler

> "The scheduler's job is to distribute ready-to-run goroutines over worker threads. … G - goroutine. M - worker thread, or machine. P - processor, a resource that is required to execute Go code."
> — [`src/runtime/proc.go`](https://go.dev/src/runtime/proc.go) header comment

This skill owns the **performance** layer of goroutines and the scheduler: what a goroutine *costs*, how the G-M-P machine turns `GOMAXPROCS` into parallelism, and the runtime knobs that decide throughput in production and in containers. The **correctness** layer — goroutine lifetime, leaks, "never start a goroutine you can't stop," `WaitGroup`/`errgroup` — lives in **`go-concurrency-goroutines`**; read it first if the question is *will this goroutine ever exit*. Pool sizing and bounded fan-out route to **`go-perf-worker-pools-throughput`**.

All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.25/1.26.

---

## 1. What a Goroutine Costs — Cheap, Not Free

A goroutine is not a thread. Since Go 1.4's contiguous stacks, "the default starting size for a goroutine's stack … has been reduced from 8192 bytes to 2048 bytes" ([Go 1.4 release notes](https://go.dev/doc/go1.4)); the runtime constant is still `stackMin = 2048` ([`src/runtime/stack.go`](https://go.dev/src/runtime/stack.go)). The stack is **not fixed**: "When a stack limit is reached, a new, larger stack is allocated, all active frames for the goroutine are copied there, and any pointers into the stack are updated" ([Go 1.4](https://go.dev/doc/go1.4)) — and it shrinks again during GC. So a goroutine starts at ~2KB and pays only for the depth it actually uses.

That makes a goroutine **cheap** — launching hundreds of thousands or millions is a supported design — but **not free**. Each one costs the stack (≥2KB live), scheduler bookkeeping, and GC scan work on its stack. The failure mode is **unbounded** spawning: one goroutine per inbound item with no ceiling means stacks and run-queue depth grow with load until the box falls over.

```go
// WRONG — unbounded fan-out: one goroutine per item, no ceiling
for _, item := range millionItems {
	go handle(item) // memory + scheduler pressure scales with input size
}
```

The fix is a **bounded** pool (`errgroup.SetLimit`, a semaphore, N workers) — sizing belongs to **`go-perf-worker-pools-throughput`**, and the lifetime/leak rules to **`go-concurrency-goroutines`**. The point here: "a goroutine is cheap" is a license for *concurrency*, not for *unbounded* concurrency.

---

## 2. The G-M-P Model

The scheduler multiplexes goroutines onto OS threads through three entities ([`proc.go`](https://go.dev/src/runtime/proc.go)):

- **G — goroutine.** A unit of work with its own stack; thousands per thread.
- **M — machine.** An OS thread. "M must have an associated P to execute Go code, however it can be blocked or in a syscall w/o an associated P" ([`proc.go`](https://go.dev/src/runtime/proc.go)).
- **P — processor.** "A resource that is required to execute Go code." The number of Ps is `GOMAXPROCS`. A P owns a **local run queue** of runnable Gs; there is also a global run queue.

Parallelism is bounded by **P**, not by the number of Gs or Ms: at most `GOMAXPROCS` goroutines run Go code simultaneously, because only that many Ps exist. When a P's local queue empties, its M **steals** work from another P's queue (or the global queue) — the work-stealing design that keeps all Ps busy without a single contended global lock ([Scalable Go Scheduler Design Doc](https://golang.org/s/go11sched), linked from `proc.go`). The practical consequence: spawning more goroutines past `GOMAXPROCS` does **not** add parallelism for CPU-bound work — it only deepens the queues the scheduler has to manage.

---

## 3. GOMAXPROCS Is the Parallelism Limit

`runtime.GOMAXPROCS` "sets the maximum number of CPUs that can be executing simultaneously" ([`runtime` pkg](https://pkg.go.dev/runtime#GOMAXPROCS)); the env var "limits the number of operating system threads that can execute user-level Go code simultaneously" ([`runtime`](https://pkg.go.dev/runtime#hdr-Environment_Variables)). Stated plainly by the 1.25 blog: "`GOMAXPROCS` is a parallelism limit. If `GOMAXPROCS=8` Go will never run more than 8 goroutines at a time" ([container-aware GOMAXPROCS](https://go.dev/blog/container-aware-gomaxprocs)).

So when someone says "my goroutines aren't running in parallel," the first checks are: is `GOMAXPROCS` > 1, and is the work actually CPU-bound (vs all blocked on one lock or one channel — an off-CPU problem the **execution tracer** diagnoses, see `go-perf-execution-tracer`). Manually pinning `GOMAXPROCS` is rarely the win it looks like — and on Go 1.25 it has a sharp side effect (§4): **setting it at all disables the container-aware default**.

`GOMAXPROCS(0)` reads the current value without changing it (passing `n < 1` "does not change the current setting" ([`runtime`](https://pkg.go.dev/runtime#GOMAXPROCS))) — useful for sizing a CPU-bound worker pool to the live parallelism limit:

```go
// Read the effective parallelism without mutating it.
procs := runtime.GOMAXPROCS(0) // e.g. the cgroup-derived value on Go 1.25
cpus := runtime.NumCPU()       // host logical CPUs — may be much larger in a container
// On Go 1.25 in a limited container, procs < cpus by design; size pools to procs.
```

---

## 4. Container-Aware GOMAXPROCS (Go 1.25)

Before Go 1.25, `GOMAXPROCS` defaulted to `runtime.NumCPU()` — the host's logical CPU count. In a container with a CPU limit (e.g. Kubernetes `resources.limits.cpu`), a binary on a 64-core node would set `GOMAXPROCS=64` while permitted far less, **oversubscribing** and getting throttled by the kernel: "Throttling … completely pauses application execution for the remainder of the throttling period. The throttling period is typically 100ms, so throttling can cause substantial tail latency impact" ([container-aware GOMAXPROCS](https://go.dev/blog/container-aware-gomaxprocs)).

**Go 1.25 fixes the default.** Two changes ([Go 1.25 release notes](https://go.dev/doc/go1.25)):

1. "On Linux, the runtime considers the CPU bandwidth limit of the cgroup containing the process, if any. If the CPU bandwidth limit is lower than the number of logical CPUs available, `GOMAXPROCS` will default to the lower limit." This corresponds to the Kubernetes "CPU limit" option (not "CPU requests").
2. "On all OSes, the runtime periodically updates `GOMAXPROCS` if the number of logical CPUs available or the cgroup CPU bandwidth limit change" — so an on-the-fly limit change is picked up automatically.

CPU limits are a *throughput* limit and can be fractional (e.g. 2.5 CPU); `GOMAXPROCS` must be a positive integer, and "Go always rounds up to enable use of the full CPU limit" ([blog](https://go.dev/blog/container-aware-gomaxprocs)).

The toggles ([Go 1.25 release notes](https://go.dev/doc/go1.25)):

- Both behaviors "are automatically disabled if `GOMAXPROCS` is set manually via the `GOMAXPROCS` environment variable or a call to `runtime.GOMAXPROCS`." This is the trap: a hard-coded `runtime.GOMAXPROCS(n)` or an `GOMAXPROCS=` env var **opts out of the container default entirely**, including the periodic updates.
- They "can also be disabled explicitly with the GODEBUG settings `containermaxprocs=0` and `updatemaxprocs=0`, respectively." With `GODEBUG=containermaxprocs=0`, `GOMAXPROCS` defaults back to `runtime.NumCPU` ([`runtime`](https://pkg.go.dev/runtime#GOMAXPROCS)).
- `runtime.SetDefaultGOMAXPROCS()` "sets `GOMAXPROCS` to the runtime default value, as if the `GOMAXPROCS` environment variable is not set" — use it to **re-enable** the container-aware default after something disabled it, or to force an immediate re-read ([`runtime`](https://pkg.go.dev/runtime#SetDefaultGOMAXPROCS)).

```go
// On Go 1.25, prefer the default — do NOT hard-code GOMAXPROCS in a container;
// pinning it opts out of the cgroup-aware default and the periodic updates.
// If a library or env var disabled it, re-enable the container-aware default:
runtime.SetDefaultGOMAXPROCS()
```

For older Go, the `uber-go/automaxprocs` package was the standard workaround; on 1.25+ it is no longer needed.

---

## 5. Syscalls and cgo Block an M, Not a P

When a goroutine makes a **blocking syscall** (or a cgo call), its M blocks in the kernel — but the runtime detaches the **P** so another M can keep running the remaining goroutines. "There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the `GOMAXPROCS` limit" ([`runtime`](https://pkg.go.dev/runtime#hdr-Environment_Variables)). This is why a server with thousands of goroutines each blocked on I/O still makes progress on `GOMAXPROCS` cores.

The cost is the **handoff**: detaching the P, and waking or spinning up a fresh M to adopt it, is real work. Code that hammers short blocking syscalls (or crosses the cgo boundary in a tight loop) pays repeated handoff overhead and can balloon the OS-thread count. Two levers: batch syscalls (see `go-perf-buffered-io`, `go-perf-zerocopy-and-syscalls`) so there are fewer of them, and cap threads with `runtime/debug.SetMaxThreads` as a safety valve. Network I/O on Go's own `net` types does **not** block an M — it parks the goroutine on the runtime poller — so this concern is specific to file/blocking syscalls and cgo.

---

## 6. Observing the Scheduler — GODEBUG=schedtrace

To see whether the scheduler is the bottleneck, read its own trace. "Setting `schedtrace=X` causes the scheduler to emit a single line to standard error every X milliseconds, summarizing the scheduler state"; adding `scheddetail=1` "causes the scheduler to emit detailed multiline info … describing state of the scheduler, processors, threads and goroutines" ([`runtime`](https://pkg.go.dev/runtime#hdr-Environment_Variables)).

```sh
GODEBUG=schedtrace=1000 ./server          # one summary line per second
GODEBUG=schedtrace=1000,scheddetail=1 ./server   # per-P / per-M / per-G detail
```

A summary line looks like this (fields are runtime-internal and version-dependent, so read trends, not absolutes):

```text
SCHED 1000ms: gomaxprocs=8 idleprocs=0 threads=21 ... runqueue=137 [12 18 22 ...]
```

Read it for **run-queue depth**: `runqueue=` is the global queue and the bracketed list is each P's local queue. A persistently deep global queue with `idleprocs=0` (all Ps busy) means you are CPU-saturated — adding goroutines won't help; you need less work or more cores. The opposite — idle Ps with a near-empty runnable count — means the goroutines are blocked **off-CPU** (§5, mistake #5), a tracer/contention question, not a `GOMAXPROCS` one. For *latency* attribution (how long a goroutine sat runnable before running) the execution tracer and `runtime/metrics` `/sched/latencies:seconds` are sharper tools — route to **`go-perf-execution-tracer`** and **`go-perf-godebug-and-metrics`**.

---

## 7. GC Assist Steals an Allocating Goroutine's CPU

Scheduler "overhead" is sometimes really GC. When the program allocates faster than the background collector keeps up, the runtime recruits the *allocating* goroutines to help: `runtime.gcAssistAlloc` is the "function goroutines enter to yield some of their time to assist the GC with scanning and marking. A large amount of cumulative time spent here (>5%) indicates that the application is likely out-pacing the GC" ([GC guide](https://go.dev/doc/gc-guide)). It shows up as "User goroutines assisting the GC in response to a high allocation rate" ([GC guide](https://go.dev/doc/gc-guide)).

The tell: a CPU profile shows time in `gcAssistAlloc`, and request latency correlates with GC cycles. The fix is **fewer allocations**, not a scheduler knob — route allocation reduction to `go-perf-allocator-internals` / `go-perf-gc-tuning` and the tail-latency framing to `go-perf-tail-latency`.

---

## 8. Goroutine-Per-Request vs Bounded

Goroutine-per-request (one goroutine per connection/request) is **idiomatic and correct** for servers — `net/http` does exactly this, and the ~2KB stacks make it scale to very high connection counts. Spawning is fine when the count is bounded by something external you trust (open connections, a fixed work set). Bound it when the count is driven by **unbounded input** (a queue you drain, a fan-out over a large slice, a recursive explosion) — there, an unbounded `go` per item is the §1 failure. The decision rule: *is there a natural ceiling on how many run at once?* If not, impose one. The sizing of that ceiling (CPU-bound near `GOMAXPROCS`, I/O-bound higher) is **`go-perf-worker-pools-throughput`**.

Watch the live count with `runtime.NumGoroutine()` (or the `/sched/goroutines:goroutines` `runtime/metrics` sample): a count that climbs with load and never drains is the tell for unbounded spawning *or* a leak — the leak case (goroutines that never exit) is `go-concurrency-goroutines`, the throughput case (too many alive at once) is the bounded-pool fix here.

---

## 9. The Headline Disciplines

| Discipline | The rule | Source |
|---|---|---|
| Cheap ≠ free | ~2KB start, but bound fan-out over unbounded input | "reduced from 8192 bytes to 2048 bytes" ([Go 1.4](https://go.dev/doc/go1.4)) |
| P bounds parallelism | At most `GOMAXPROCS` Gs run Go code at once | "GOMAXPROCS is a parallelism limit" ([blog](https://go.dev/blog/container-aware-gomaxprocs)) |
| Trust the 1.25 default | Don't pin `GOMAXPROCS` in a container; it opts out | "automatically disabled if GOMAXPROCS is set manually" ([Go 1.25](https://go.dev/doc/go1.25)) |
| Re-enable explicitly | `SetDefaultGOMAXPROCS()` restores the container default | "as if the GOMAXPROCS environment variable is not set" ([runtime](https://pkg.go.dev/runtime#SetDefaultGOMAXPROCS)) |
| Syscalls block M, not P | Blocked-in-syscall threads don't count against the limit | "no limit to the number of threads … blocked in system calls" ([runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables)) |
| Measure the scheduler | `schedtrace` before any knob | "emit a single line … summarizing the scheduler state" ([runtime](https://pkg.go.dev/runtime#hdr-Environment_Variables)) |
| Assist ≠ scheduler | `gcAssistAlloc` in a profile is allocation pressure | ">5% … out-pacing the GC" ([GC guide](https://go.dev/doc/gc-guide)) |

---

## 10. Don't

- **Don't spawn one goroutine per item over unbounded input.** Cheap ≠ free; bound the fan-out (`go-perf-worker-pools-throughput`); lifetime/leak rules in `go-concurrency-goroutines`.
- **Don't hard-code `runtime.GOMAXPROCS(n)` (or set the env var) in a container on Go 1.25** — it disables the cgroup-aware default *and* the periodic updates; prefer the default or `SetDefaultGOMAXPROCS()` ([Go 1.25](https://go.dev/doc/go1.25)).
- **Don't assume `GOMAXPROCS` = host cores under a CPU limit on pre-1.25 Go** — it oversubscribes and the kernel throttles in ~100ms bursts ([blog](https://go.dev/blog/container-aware-gomaxprocs)); upgrade or use `automaxprocs`.
- **Don't expect more goroutines than `GOMAXPROCS` to add parallelism** for CPU-bound work — they only deepen run queues ([blog](https://go.dev/blog/container-aware-gomaxprocs)).
- **Don't blame the scheduler for an off-CPU stall** — goroutines blocked on a lock/channel are a tracer/contention question (`go-perf-execution-tracer`, `go-perf-block-mutex-profiles`), not a `GOMAXPROCS` one.
- **Don't ignore `gcAssistAlloc` in a CPU profile** — that "scheduler overhead" is GC pressure; reduce allocations (`go-perf-gc-tuning`).
- **Don't hammer short blocking syscalls/cgo in a tight loop** — each pays P-handoff and can grow the thread count; batch instead (`go-perf-buffered-io`).

---

## 11. Routing to Related Skills

- `go-concurrency-goroutines` — goroutine **correctness**: lifetime, leaks, defined exit, `WaitGroup`/`wg.Go`, `errgroup`, synchronous libraries. Read first for "will this exit?".
- `go-perf-worker-pools-throughput` — bounding fan-out and **sizing** pools (`SetLimit`, semaphore, batching).
- `go-perf-execution-tracer` — scheduler-latency and goroutine-analysis views; the flight recorder for capturing a stall.
- `go-perf-godebug-and-metrics` — the GODEBUG catalog and `runtime/metrics` (`/sched/goroutines`, `/sched/latencies:seconds`).
- `go-perf-tail-latency` — p99/p999 and why throttling/GC drive the tail.
- `go-perf-gc-tuning` / `go-perf-allocator-internals` — reducing the allocation rate behind GC assist.
- `go-perf-atomics-vs-locks`, `go-perf-contention-and-sharding` — when "not parallel" is really lock contention.
- `go-version-feature-map` — version floors: container-aware `GOMAXPROCS` 1.25.

---

## 12. Reference Files

Scheduler/goroutine performance anti-patterns in LLM-generated Go, each with wrong/right contrast and citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)