# Go Perf Buffered Io

> Guides buffered I/O in Go — wrapping os.File/net.Conn in bufio so small reads/writes batch into few large syscalls, the mandatory Flush+error check, bufio.Scanner's 64KB token limit and Scanner.Buffer, io.Copy's ReaderFrom/WriterTo fast paths and io.CopyBuffer, and streaming vs loading whole files. Fires on "buffer this I/O", "too many syscalls", "bufio", "scanner token too long", "copy a file efficiently", "slow file writing". Routes sendfile/splice to go-perf-zerocopy-and-syscalls, bytes to go-perf-strings-bytes-zerocopy.

- Skill: `ctoth/go-perf-buffered-io` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-buffered-io`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-buffered-io/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-buffered-io

---


# Go Buffered I/O

The measure-first discipline — **profile or benchmark → change one thing → measure again** — belongs to **`go-performance`** and **`go-perf-methodology`**; read them first. This skill owns one lever they leave on the table: **buffering** raw I/O so many small `Read`/`Write` calls become few large syscalls, plus the `io.Copy` fast paths and the whole-file-vs-streaming decision.

All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.25 / 1.26.

---

## 1. Why Buffer — Each Raw Call Is a Syscall

A raw `*os.File` is a thin wrapper over the kernel: `Write` "writes len(b) bytes from b to the File … returns a non-nil error when n != len(b)" and `Read` "reads up to len(b) bytes from the File" ([os](https://pkg.go.dev/os#File.Write)) — each call crosses into the kernel. Writing a log line a field at a time, or reading a byte at a time, pays one syscall per call: **the #1 I/O performance mistake is unbuffered small writes.**

`bufio` exists to fix exactly this: it "implements buffered I/O. It wraps an `io.Reader` or `io.Writer` object, creating another object (Reader or Writer) that also implements the interface but provides buffering" ([bufio](https://pkg.go.dev/bufio)). A `bufio.Writer` accumulates small writes in memory and flushes them to the underlying writer in one large syscall when the buffer fills; a `bufio.Reader` reads a big chunk once and serves small reads from memory. Many small ops → few large syscalls → far less CPU spent in kernel transitions.

```go
// WRONG: one write(2) syscall per line.
for _, rec := range records {
	fmt.Fprintln(f, rec) // f is *os.File
}

// RIGHT: writes batched into the buffer, few syscalls.
w := bufio.NewWriter(f)
for _, rec := range records {
	fmt.Fprintln(w, rec)
}
if err := w.Flush(); err != nil { // see §2 — mandatory
	return err
}
```

---

## 2. `bufio.Writer` — You MUST `Flush` (and Check the Error)

A `bufio.Writer` holds buffered bytes that have **not yet reached the underlying writer**. `Flush` "writes any buffered data to the underlying `io.Writer`" ([bufio](https://pkg.go.dev/bufio#Writer.Flush)). If the program returns or the file closes while the buffer is non-empty, **those bytes are silently lost** — a truncated file with no error anywhere. This is the second-most-common bufio bug after not buffering at all.

Two rules:

- **Always `Flush` before the writer goes out of scope**, and do it *before* closing the underlying file (a bare `defer f.Close()` runs after the function returns but does nothing about un-flushed bufio bytes).
- **Check the `Flush` error.** `bufio.Writer` defers all I/O errors: "If an error occurs writing to a Writer, no more data will be accepted and all subsequent writes, and `Writer.Flush`, will return the error" ([bufio](https://pkg.go.dev/bufio#Writer)). A failing disk write may surface **only** at `Flush` — `w.Write(...)` can return nil while the real error waits in the buffer.

```go
func writeAll(f *os.File, recs []string) (err error) {
	w := bufio.NewWriter(f)
	for _, r := range recs {
		if _, err = w.WriteString(r); err != nil {
			return err
		}
	}
	return w.Flush() // surfaces any deferred write error; never ignore it
}
```

`defer w.Flush()` discards the error — acceptable only when an earlier explicit `Flush` already ran, or via a named-return wrapper (`defer func() { if e := w.Flush(); err == nil { err = e } }()`).

---

## 3. `bufio.Reader` and Sizing the Buffer

`NewReader`/`NewWriter` give "a new Reader/Writer whose buffer has the default size" ([bufio](https://pkg.go.dev/bufio#NewReader)) — that default is **4096 bytes** (`defaultBufSize` in `src/bufio/bufio.go`). For bulk streaming this is often too small; reading a multi-MB file 4 KiB at a time is still thousands of syscalls.

Size the buffer to the workload with `NewReaderSize`/`NewWriterSize`, which return a reader/writer "whose buffer has **at least** the specified size" ([bufio](https://pkg.go.dev/bufio#NewReaderSize)). A 64 KiB–1 MiB buffer for large sequential transfers cuts syscall count by orders of magnitude; tiny interactive protocols want the small default. Don't guess — benchmark the throughput (route to `go-perf-benchmarking-statistics`). Note both `NewReaderSize` and `NewWriterSize` **reuse** the argument if it is already a `bufio` reader/writer of sufficient size, so double-wrapping is cheap, not doubly buffered.

The same applies to a `net.Conn`: a chatty protocol that writes a header, then a body, then a trailer as three `conn.Write` calls is three packets / three syscalls. Wrap once and flush per logical message:

```go
bw := bufio.NewWriterSize(conn, 16*1024)
writeHeader(bw)
writeBody(bw)
writeTrailer(bw)
if err := bw.Flush(); err != nil { // one syscall, often one segment
	return err
}
```

For a stream that needs both directions, `bufio.NewReadWriter(r, w)` pairs a `*Reader` and `*Writer` into one value — but it is just a struct holding the two; **you still `Flush` the writer half yourself.**

---

## 4. `bufio.Scanner` and the 64 KB Token Limit

`Scanner` "provides a convenient interface for reading data such as a file of newline-delimited lines of text … the default split function breaks the input into lines" ([bufio](https://pkg.go.dev/bufio#Scanner)). The trap: a single token (e.g. one very long line, or a JSON blob with `bufio.ScanLines`) **cannot exceed the buffer cap**. The default cap is `MaxScanTokenSize`, documented as `= 64 * 1024` ([bufio](https://pkg.go.dev/bufio#pkg-constants)).

When a token is too big, scanning **stops unrecoverably** — "Scanning stops unrecoverably at EOF, the first I/O error, or a token too large to fit in the `Scanner.Buffer`" ([bufio](https://pkg.go.dev/bufio#Scanner.Scan)). `Scan` returns `false` and `Err()` returns `ErrTooLong` — `errors.New("bufio.Scanner: token too long")` ([bufio](https://pkg.go.dev/bufio#pkg-variables)). It does **not** panic, so a loop that ignores `Err()` silently processes a truncated prefix of the input — a correctness bug, not just a slow one. **Always check `scanner.Err()` after the loop.**

Raise the limit with `Scanner.Buffer`, which "sets the initial buffer to use when scanning and the maximum size of buffer that may be allocated during scanning" ([bufio](https://pkg.go.dev/bufio#Scanner.Buffer)):

```go
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) // start 64KiB, grow to 4MiB max
for sc.Scan() {
	process(sc.Bytes()) // Bytes() is only valid until the next Scan
}
if err := sc.Err(); err != nil { // catches ErrTooLong and I/O errors
	return err
}
```

For genuinely unbounded lines, don't pick a huge cap — use `bufio.Reader.ReadString('\n')` / `ReadLine` and stream, so one pathological line can't force a multi-MB allocation. `sc.Bytes()` aliases the internal buffer; copy it (`append([]byte(nil), sc.Bytes()...)` or `sc.Text()`) if you retain it past the next `Scan`.

---

## 5. `io.Copy` and the `ReaderFrom`/`WriterTo` Fast Paths

To move all of one stream into another, reach for `io.Copy`, not a hand-rolled `Read`/`Write` loop. Beyond being shorter, it auto-selects a fast path: "If src implements `WriterTo`, the copy is implemented by calling `src.WriteTo(dst)`. Otherwise, if dst implements `ReaderFrom`, the copy is implemented by calling `dst.ReadFrom(src)`" ([io](https://pkg.go.dev/io#Copy)). Those interfaces let a type move data "without … a buffer" of your own — `WriterTo.WriteTo` "writes data to w until there's no more data to write," `ReaderFrom.ReadFrom` "reads data from r until EOF" ([io](https://pkg.go.dev/io#ReaderFrom)). This is the hook that lets file→socket copies drop into the kernel `sendfile`/`splice` path (depth: **`go-perf-zerocopy-and-syscalls`**).

When **neither** interface applies, `io.Copy` allocates one internal 32 KiB buffer (`src/io/io.go`) and loops. If you copy repeatedly, reuse one buffer with `io.CopyBuffer`, which "is identical to Copy except that it stages through the provided buffer (if one is required) rather than allocating a temporary one" ([io](https://pkg.go.dev/io#CopyBuffer)). Two caveats from the same doc:

- It **panics on a zero-length buffer**: "if buf is nil, one is allocated; otherwise if it has zero length, CopyBuffer panics" — pass `make([]byte, 32*1024)`, never `[]byte{}` ([io](https://pkg.go.dev/io#CopyBuffer)).
- The buffer is **ignored when a fast path exists**: "If either src implements WriterTo or dst implements ReaderFrom, buf will not be used" ([io](https://pkg.go.dev/io#CopyBuffer)) — so don't wrap an `*os.File` in `bufio` *just* to feed `io.Copy`; the wrapper hides the `ReaderFrom`/`WriterTo` method set and can **disable** the zero-copy fast path.

```go
// Reuse one buffer across many copies; check the error.
buf := make([]byte, 64*1024)
for _, job := range jobs {
	if _, err := io.CopyBuffer(job.dst, job.src, buf); err != nil {
		return err
	}
}
```

---

## 6. Whole-File vs Streaming — Don't Load a Huge File Into RAM

For a small, bounded file, `os.ReadFile` is correct and clear: it "reads the named file and returns the contents … Because ReadFile reads the whole file, it does not treat an EOF from Read as an error" ([os](https://pkg.go.dev/os#ReadFile)). But it returns the **entire file as one `[]byte`** — on a multi-GB file or untrusted/unbounded input that is an OOM, not an optimization.

The same trap applies to `io.ReadAll`, which "reads from r until an error or EOF" ([io](https://pkg.go.dev/io#ReadAll)) — fine for a known-small body, a memory bomb on an unbounded reader (cap it with `io.LimitReader` first). For large files, **stream**: wrap in `bufio.NewReaderSize` and process incrementally, or `io.Copy` straight to the destination so peak memory stays at one buffer regardless of file size. To write a whole small buffer at once use `os.WriteFile` (note: it "requires multiple system calls … a failure mid-operation can leave the file in a partially written state" ([os](https://pkg.go.dev/os#WriteFile))).

Rule of thumb: **bounded and small → `os.ReadFile`/`os.WriteFile`; large or unbounded → buffered stream / `io.Copy`.**

---

## 7. Don't

- **Don't issue small unbuffered writes/reads on an `*os.File` or `net.Conn`** — one syscall each; wrap in `bufio` ([bufio](https://pkg.go.dev/bufio)).
- **Don't forget `bufio.Writer.Flush`** — un-flushed bytes are silently lost, producing a truncated file with no error ([bufio](https://pkg.go.dev/bufio#Writer.Flush)).
- **Don't ignore the `Flush` error** — `bufio.Writer` defers write errors to `Flush`, so a failed disk write surfaces only there ([bufio](https://pkg.go.dev/bufio#Writer)).
- **Don't ignore `Scanner.Err()`** — on a token over 64 KB, `Scan` returns false with `ErrTooLong` and the rest of the input is silently dropped; raise the cap with `Scanner.Buffer` ([bufio](https://pkg.go.dev/bufio#Scanner.Buffer)).
- **Don't pass a zero-length buffer to `io.CopyBuffer`** — it panics; pass `nil` or a sized buffer ([io](https://pkg.go.dev/io#CopyBuffer)).
- **Don't `bufio`-wrap a file solely to feed `io.Copy`** — it hides `ReaderFrom`/`WriterTo` and can disable the kernel sendfile fast path ([io](https://pkg.go.dev/io#Copy)).
- **Don't `os.ReadFile`/`io.ReadAll` an unbounded or huge input** — stream it instead, or cap with `io.LimitReader` ([os](https://pkg.go.dev/os#ReadFile)).
- **Don't allocate your own copy buffer when `io.Copy` would** — let it pick the fast path; reuse one buffer via `io.CopyBuffer` only across repeated copies ([io](https://pkg.go.dev/io#CopyBuffer)).

---

## 8. Routing to Related Skills

**This plugin (`go-perf-*` depth):**

- `go-perf-zerocopy-and-syscalls` — kernel `sendfile`/`splice` (the `ReaderFrom`/`WriterTo` payoff), `net.Buffers` writev, mmap, batching syscalls.
- `go-perf-strings-bytes-zerocopy` — building strings/bytes without copies (`strings.Builder`, `strconv.Append*`, `unsafe.String`/`Slice`).
- `go-perf-sync-pool` — pooling and reusing the copy/scan buffers this skill sizes.
- `go-perf-benchmarking-statistics` — proving a buffer-size change is a real win, not noise.
- `go-perf-methodology` — the "should I optimize this I/O at all" judgment.

**Parent / sibling marketplace (`go-*` correctness):**

- `go-performance` — the measure-first overview this skill deepens.
- `go-strings-bytes-runes` — string/byte/rune correctness behind the perf moves.

---

## 9. Reference Files

Buffered-I/O anti-patterns in LLM-generated Go, each with wrong/right contrast and citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)