Go Perf Contention And Sharding

Go Contention & Sharding

"The Map type is optimized for two common use cases ... In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex." — pkg.go.dev/sync

A single mutex guarding shared state is a serialization point: while one goroutine holds it, every other core that wants that state stalls, so throughput stops scaling and can fall as cores are added. This skill owns the performance fix — splitting one lock into many so independent operations rarely collide. The correctness rules of the primitives — sync.Map's two cases, the no-copy Mutex rule, typed atomics — belong to go-sync-primitives; read it first. Always prove the contention exists before sharding: a block/mutex profile (go-perf-block-mutex-profiles) tells you which lock is hot, and benchstat (go-perf-benchmarking-statistics) tells you the shard actually helped.

All Go below builds clean under gofmt, go vet, and go test on Go 1.26.

1. The Core Idea — One Hot Lock Serializes Every Core

When a profile shows goroutines spending wall-clock time waiting on one Mutex, the lock — not the work it guards — is the bottleneck. The CPU profile is blind to this (it only sees on-CPU time); the mutex profile records it, but is off by default and records nothing until you call runtime.SetMutexProfileFraction — "controls the fraction of mutex contention events that are reported in the mutex profile ... To turn off profiling entirely, pass rate 0" (runtime) — and runtime.SetBlockProfileRate for the block profile (runtime). Enablement and reading these are owned by go-perf-block-mutex-profiles.

The fix is lock striping: replace the one lock + one map with N shards, each its own lock guarding its own slice of the keyspace. Hash the key to a shard; two goroutines touching keys in different shards never contend. Contention drops roughly N× when keys spread evenly, because at most ~1/N of the traffic lands on any one lock.

2. A Sharded Map — Lock Striping by Key Hash

The pattern: a fixed array of shards, each a Mutex (or RWMutex when reads dominate — see go-perf-atomics-vs-locks) over its own map, selected by hashing the key. This is exactly how orcaman/concurrent-map works — it shards "the map with minimal time spent waiting for locks," and is pitched for "something more like in-memory db" with frequent writes, unlike sync.Map which it notes "is designed for append-only scenarios" (orcaman/concurrent-map).

// shardCount is fixed and a power of two so key%shardCount is a mask, not a divide.
// orcaman/concurrent-map ships SHARD_COUNT = 32 as its default ([orcaman/concurrent-map]).
const shardCount = 32

type shard struct {
	mu sync.RWMutex
	m  map[string]int
	// Pad so two shards never share a 64-byte cache line; see go-perf-false-sharing.
	_ [64 - (unsafe.Sizeof(sync.RWMutex{})+unsafe.Sizeof(map[string]int(nil)))%64]byte
}

type ShardedMap struct{ shards [shardCount]shard }

func NewShardedMap() *ShardedMap {
	s := &ShardedMap{}
	for i := range s.shards {
		s.shards[i].m = make(map[string]int)
	}
	return s
}

// shard() spreads keys with a real hash — FNV-1a, as orcaman/concurrent-map does (full
// fnv32 in references/common-mistakes.md) — then masks to a shard since shardCount is a power of two.
func (s *ShardedMap) shard(key string) *shard { return &s.shards[fnv32(key)%shardCount] }

func (s *ShardedMap) Load(key string) (int, bool) {
	sh := s.shard(key)
	sh.mu.RLock()
	defer sh.mu.RUnlock()
	v, ok := sh.m[key]
	return v, ok
}

func (s *ShardedMap) Store(key string, v int) {
	sh := s.shard(key)
	sh.mu.Lock()
	sh.m[key] = v
	sh.mu.Unlock()
}

Two failure modes: a bad hash that clusters keys recreates a single hot shard (use a real hash, not len(key) or a sequential id's low bits); and unpadded shards sharing a cache line, so independent locks still bounce the line between cores — false sharing, owned by go-perf-false-sharing.

3. Shard Count — ≈ GOMAXPROCS or a Power of Two

The shard count sets the contention ceiling. Guidance:

• Floor it at GOMAXPROCS. With fewer shards than cores, cores still queue on shared locks. A small multiple of GOMAXPROCS (or a fixed power of two comfortably above it, like orcaman's 32) leaves headroom as the schedulable parallelism grows. Note Go 1.25 makes GOMAXPROCS container-aware (it reads the cgroup CPU limit), so the effective core count under a CPU quota may be lower than the host's (Go 1.25).
• Prefer a power of two, so hash % shardCount lowers to a bitmask instead of a division on the hot path.
• Don't over-shard. Each shard is a live map with its own bucket overhead; thousands of shards waste memory and cache for no contention win past the core count. And pad each shard to a cache line, or adjacent shard locks false-share — route to go-perf-false-sharing.

4. sync.Map vs a Sharded Map — Match the Access Pattern

sync.Map is not a general "concurrent map" — its own doc is explicit: "The Map type is specialized. Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content" (sync). It is "optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention" (sync). The decision (its API/correctness depth lives in go-sync-primitives):

For the general read-write case a sharded plain map is usually faster and type-safe (no any boxing, no per-read type assertion) — and it lets you hold an invariant across the map under one shard's lock, which sync.Map cannot. Reach for sync.Map only when the pattern is genuinely one of its two documented cases.

5. Counters — Stripe and Sum, Don't Hammer One Atomic

A single atomic.Int64 incremented by every core is itself a contention point: each Add must win exclusive ownership of that one cache line, so the line ping-pongs between cores under load. The fix mirrors map sharding — N per-core counter cells, each on its own cache line, summed on read:

type cell struct {
	n atomic.Int64
	_ [64 - 8]byte // one cell per cache line; padding owned by go-perf-false-sharing
}

type StripedCounter struct{ cells [shardCount]cell }

// id is any cheap per-goroutine spreader (a worker index, a hashed key).
func (c *StripedCounter) Add(id uint64, delta int64) { c.cells[id%shardCount].n.Add(delta) }

func (c *StripedCounter) Sum() int64 {
	var total int64
	for i := range c.cells {
		total += c.cells[i].n.Load()
	}
	return total
}

This trades an exact snapshot read (Sum is not instantaneously consistent) for write throughput that scales with cores — the right trade for high-frequency metrics. The atomic-vs-mutex cost model is go-perf-atomics-vs-locks; padding is go-perf-false-sharing.

6. Batch Under the Lock — Amortize the Acquisition

Lock acquisition has a fixed cost paid per acquisition. Acquiring once per item turns N items into N lock/unlock round-trips; acquiring once for the whole batch pays it once.

// WRONG — one lock acquisition per item; the lock cost dominates the loop
for _, item := range items {
	mu.Lock()
	queue = append(queue, item)
	mu.Unlock()
}

// RIGHT — one acquisition amortized over the whole batch
mu.Lock()
queue = append(queue, items...)
mu.Unlock()

Hold the lock only for the shared mutation; do per-item computation before Lock so the critical section stays short (the small-critical-section rule is go-sync-primitives). Batching and lock striping compose: shard first, then batch within each shard's lock.

7. singleflight — Collapse Duplicate Concurrent Work

When many goroutines independently request the same expensive thing at once (a cache miss stampede, a thundering herd on one DB row), the contention is duplicated work, not a lock. golang.org/x/sync/singleflight provides "a duplicate function call suppression mechanism": Group.Do "executes and returns the results of the given function, making sure that only one execution is in-flight for a given key at a time. If a duplicate comes in, the duplicate caller waits for the original to complete and receives the same results" (singleflight).

var g singleflight.Group

func (c *Cache) Fetch(ctx context.Context, key string) (*Row, error) {
	v, err, _ := g.Do(key, func() (any, error) {
		return loadFromDB(ctx, key) // exactly one DB load per key, however many callers race
	})
	if err != nil {
		return nil, err
	}
	return v.(*Row), nil
}

The third return, shared, "indicates whether v was given to multiple callers"; DoChan gives a select-able channel result, and Forget drops a key so the next call re-runs rather than reusing a stale in-flight result (singleflight). One caveat: Do couples callers — a slow or failed shared call delays everyone waiting on that key.

8. Don't

• Don't shard before a profile proves the lock is hot. Enable the mutex profile (SetMutexProfileFraction, off by default (runtime)) and confirm the wait — depth in go-perf-block-mutex-profiles.
• Don't assume sync.Map is faster. It is "specialized" and for the general read-write case a sharded map+Mutex usually wins (sync).
• Don't leave shards unpadded. Adjacent shard locks share a cache line and false-share; pad to a cache line (go-perf-false-sharing).
• Don't shard on a weak hash. len(key) or sequential-id low bits cluster keys into one hot shard; use a real hash like FNV-1a (orcaman/concurrent-map).
• Don't hammer one global atomic counter from every core, or take the lock per item — stripe the counter (§5) and batch under one acquisition (§6).
• Don't let a stampede duplicate work that singleflight would collapse (singleflight).
• Don't reach for a channel-as-mutex to "fix" contention where sharding is the real lever — it imports a channel's full failure surface to do a lock's job (go-sync-primitives).

9. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-block-mutex-profiles — measure the contention first: enabling and reading block/mutex/goroutine profiles.
• go-perf-atomics-vs-locks — the atomic-vs-Mutex / RWMutex cost model behind a shard's lock and a striped counter's cell.
• go-perf-false-sharing — padding shards and counter cells to a cache line (cpu.CacheLinePad); the non-scaling symptom.
• go-perf-maps — single-map performance (Swiss Tables, prealloc, key cost) that each shard inherits.
• go-perf-benchmarking-statistics — proving the sharded version is actually faster (n≥10, benchstat, b.RunParallel).

Parent / sibling marketplace (go-* correctness):

• go-sync-primitives — sync.Map's two cases, the no-copy Mutex/atomic rules, small critical sections, sync.Pool — the API and correctness this skill's perf depth sits on.
• go-race-and-memory-model — why unsynchronized shard access is still a data race; happens-before.
• go-version-feature-map — version floors (container-aware GOMAXPROCS 1.25; typed atomics 1.19).

10. Reference Files

Contention/sharding anti-patterns in LLM-generated Go, each with wrong/right code and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml