Buffer Manager

buffer-manager — the shared-memory page cache

Every read of any heap or index page in PG goes through the buffer manager. It maintains an in-memory cache (sized by shared_buffers) of 8 KB pages, indexed by (rlocator, forknum, blocknum). Pins reserve a buffer for a caller; refcount → 0 lets the clock-sweep evictor pick it as a replacement candidate.

Getting pin/unpin discipline right is critical — leaks cause WARNING: buffer refcount leak and eventually OOM.

The file map

The buffer pool

A fixed-size array of NBuffers (from shared_buffers) buffers in shared memory. Each buffer is:

• A BufferDesc — the metadata (state, pin count, etc.).
• The buffer page — 8 KB of actual data.

Buffers are addressed by buffer number (1-indexed into the pool). NEGATIVE numbers refer to LOCAL buffers (temp tables).

The BufferDesc state

Each BufferDesc holds packed state bits:

• BM_LOCKED — someone holds the internal spinlock.
• BM_DIRTY — page has changes not yet flushed.
• BM_VALID — the page content is loaded (not just an allocated slot).
• BM_TAG_VALID — the (rlocator, forknum, blocknum) tag is set.
• BM_IO_IN_PROGRESS — a read or write is happening; wait via CV.
• BM_IO_ERROR — the I/O failed; retry may be needed.
• BM_JUST_DIRTIED — was dirty AND has been written since; needs re-dirty check.
• Refcount — number of pins.
• Usage count — clock-sweep priority (0-5).
• Wait backend PID — for pin-waiting.

All state changes are atomic via pg_atomic_* since PG 12.

Pin / unpin

Buffer buf = ReadBuffer(rel, blocknum);  // + pin
LockBuffer(buf, BUFFER_LOCK_SHARE);      // + share content lock
/* ... use page ... */
UnlockReleaseBuffer(buf);                // - lock, - pin

The two-part lifetime: pin (reservation — prevents eviction) + content lock (SHARE / EXCLUSIVE — prevents concurrent modification). Both can be held OR released independently, though a common shortcut releases both.

Refcount management is via ResourceOwner — a subtransaction rollback releases all its pins. See resource-owners skill.

Clock-sweep replacement

When a caller needs a buffer for a NEW page and the free list is empty:

1. StrategyGetBuffer runs the clock sweep starting at the last position.
2. Each buffer visited: if refcount > 0, skip. If usage_count > 0, decrement + continue. If usage_count = 0 AND refcount = 0, pick it.
3. If the picked buffer is dirty, flush it (may block on I/O).
4. Return the pinned + free buffer.

Usage count is incremented on every ReleaseBuffer up to a cap. New buffers start at 1. The algorithm approximates LRU without the strict-LRU overhead of maintaining a linked list.

BufferAccessStrategy — the ring buffer pattern

For scans that would otherwise trash the whole buffer cache (VACUUM, big SELECTs, bulk-load COPY), the code uses a BufferAccessStrategy — a small ring of buffers that keep getting reused instead of evicting other backends' hot buffers.

Types (in include/access/heapam.h):

• BAS_NORMAL — no strategy; use the whole pool.
• BAS_BULKREAD — for full-table SELECTs. ~256 KB ring.
• BAS_BULKWRITE — for COPY / CREATE TABLE AS. Bigger ring, 16 MB.
• BAS_VACUUM — VACUUM. Distinct ring so vacuum doesn't evict user data.

A callable gets a strategy from GetAccessStrategy(BAS_...). Passes it to ReadBufferExtended — the manager uses ring slots first.

The scenario add-new-buffer-strategy covers extending this.

Local buffers (temp tables)

Temp tables + WITHOUT LOGGING relations use LOCAL buffers, not shared. In localbuf.c:

• Per-backend, in the backend's local memory.
• No shmem contention.
• Not visible to other backends (correct — temp tables are backend-private).
• Not flushed to disk on WITH ... transactions (unlogged); flushed on TEMP-TABLE with ON COMMIT PRESERVE.

Synchronous writer coordination

• bgwriter — background writer. Scans the buffer pool for dirty buffers, writes them out to smooth checkpoint I/O.
• checkpointer — the checkpointer process — the OWNER of the sync writes. Flushes ALL dirty buffers at checkpoint.
• Backends — will FLUSH a buffer on eviction if it's dirty.

The three-way race is coordinated via LATCH_WAIT + spinlocks on individual BufferDescs.

Common patch shapes

Add a new BufferAccessStrategy ring

Scenario add-new-buffer-strategy covers this. Short:

• New enum value in include/access/heapam.h (BAS_...).
• Ring size + policy in freelist.c (StrategyGetBuffer's strategy dispatch).
• Caller passes new strategy via GetAccessStrategy.

Debug "buffer refcount leak WARNING"

• Log identifies the buffer + scope.
• Common cause: code path pins buffer, early-exits without releasing.
• ResourceOwner is the safety net (releases on scope exit) but shouldn't be relied on — should ReleaseBuffer explicitly.
• PIN_HINT can also be misread — hint bits alone aren't a pin.

Change shared_buffers sizing recommendations

Not really a code change — but understanding: memory usage is roughly NBuffers * 8KB for pages plus BufferDesc structs (~80 bytes each). Adjust shared_buffers for the workload; ~25% of system memory is a common rule.

Extend BufferDesc state bits

Very rare, dangerous. Requires:

• Coordinated update of pg_atomic_* operations.
• Backward-compat: never reuse a bit position without on-disk file format bump.
• Audit every state-checking site.

Pitfalls

• Pin leak on error paths — a code path that pins then encounters ereport(ERROR) needs to be inside a ResourceOwner scope or explicit PG_TRY.
• Buffer content lock ordering — always acquire in a defined order (typically block-number order) to avoid deadlocks. See relevant per-AM code for the discipline.
• DIRTY without LOCK EXCLUSIVE is wrong — modifying a buffer's content requires EXCLUSIVE. Share-lock + modify is a classic bug source.
• MarkBufferDirty vs MarkBufferDirtyHint — the former is for WAL-logged changes; the latter is for hint-bit-style optimistic updates. Hint updates are lost on crash by design.
• Local buffers aren't in shmem — code that assumes "all buffers are in the shared pool" misses temp-table buffers. Handle both sides in bufmgr consumers.
• Ring buffers can starve — if a scan is much bigger than the ring, it constantly re-evicts its own pages. The ring is a trade-off, not a solution for arbitrary sizes.
• Clock-sweep prefers old data — if shared_buffers is undersized for the workload, clock-sweep will churn constantly. pg_stat_bgwriter + pg_buffercache show the pressure.
• pg_buffercache contrib is O(N) — scanning shared_buffers shows every buffer. On huge systems this can pause query dispatch.
• SyncOneBuffer returns early on eviction race — a caller assuming "sync happened" needs to check the return value.
• BufferPin can outlive its content lock — locking behavior split by design; consumers must understand the difference.

Related corpus

• Idioms (multiple hits): hint-bits-setbufferdirty (hint updates), spinlock-discipline (bufhdrspinlock semantics), gin-fastupdate-pending (fastupdate uses a special buffer), wal-buffer-state (WAL buffers — sibling system, not shared), xlog-region-replay (XLogReadBufferForRedo interaction).
• Subsystems: storage-buffer (this skill's home), storage-ipc (LWLocks used for BM_IO_IN_PROGRESS waits), access-heap (primary caller).
• Data structure: bufferdesc-state (definitive BufferDesc struct doc).
• Scenario: add-new-buffer-strategy (the ring-buffer patch shape).
• README: source/src/backend/storage/buffer/README — authoritative design doc; shorter than this skill.

Corpus-chain shortcut

python3 scripts/corpus-chain.py --scenario add-new-buffer-strategy
python3 scripts/corpus-chain.py --file src/backend/storage/buffer/bufmgr.c

Boundary

Use this skill for bufmgr.c internals + BufferAccessStrategy + pin/unpin discipline.

Don't use for:

• storage/smgr/ — the storage-media layer BELOW bufmgr. Different concerns.
• AIO / read_stream — sits BETWEEN bufmgr and smgr in the read path. See aio-readstream.
• Visibility map / freespace map — separate forks; use free-space-map etc.
• pg_buffercache — contrib module for introspection.
• WAL buffers — separate system (WAL is written to a different shmem area). See wal-buffer-state idiom.