Go Perf Buffered Io

Go Buffered I/O

The measure-first discipline — profile or benchmark → change one thing → measure again — belongs to go-performance and go-perf-methodology; read them first. This skill owns one lever they leave on the table: buffering raw I/O so many small Read/Write calls become few large syscalls, plus the io.Copy fast paths and the whole-file-vs-streaming decision.

All Go below builds clean under gofmt, go vet, and go test on Go 1.25 / 1.26.

1. Why Buffer — Each Raw Call Is a Syscall

A raw *os.File is a thin wrapper over the kernel: Write "writes len(b) bytes from b to the File … returns a non-nil error when n != len(b)" and Read "reads up to len(b) bytes from the File" (os) — each call crosses into the kernel. Writing a log line a field at a time, or reading a byte at a time, pays one syscall per call: the #1 I/O performance mistake is unbuffered small writes.

bufio exists to fix exactly this: it "implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or Writer) that also implements the interface but provides buffering" (bufio). A bufio.Writer accumulates small writes in memory and flushes them to the underlying writer in one large syscall when the buffer fills; a bufio.Reader reads a big chunk once and serves small reads from memory. Many small ops → few large syscalls → far less CPU spent in kernel transitions.

// WRONG: one write(2) syscall per line.
for _, rec := range records {
	fmt.Fprintln(f, rec) // f is *os.File
}

// RIGHT: writes batched into the buffer, few syscalls.
w := bufio.NewWriter(f)
for _, rec := range records {
	fmt.Fprintln(w, rec)
}
if err := w.Flush(); err != nil { // see §2 — mandatory
	return err
}

2. bufio.Writer — You MUST Flush (and Check the Error)

A bufio.Writer holds buffered bytes that have not yet reached the underlying writer. Flush "writes any buffered data to the underlying io.Writer" (bufio). If the program returns or the file closes while the buffer is non-empty, those bytes are silently lost — a truncated file with no error anywhere. This is the second-most-common bufio bug after not buffering at all.

Two rules:

• Always Flush before the writer goes out of scope, and do it before closing the underlying file (a bare defer f.Close() runs after the function returns but does nothing about un-flushed bufio bytes).
• Check the Flush error. bufio.Writer defers all I/O errors: "If an error occurs writing to a Writer, no more data will be accepted and all subsequent writes, and Writer.Flush, will return the error" (bufio). A failing disk write may surface only at Flush — w.Write(...) can return nil while the real error waits in the buffer.

func writeAll(f *os.File, recs []string) (err error) {
	w := bufio.NewWriter(f)
	for _, r := range recs {
		if _, err = w.WriteString(r); err != nil {
			return err
		}
	}
	return w.Flush() // surfaces any deferred write error; never ignore it
}

defer w.Flush() discards the error — acceptable only when an earlier explicit Flush already ran, or via a named-return wrapper (defer func() { if e := w.Flush(); err == nil { err = e } }()).

3. bufio.Reader and Sizing the Buffer

NewReader/NewWriter give "a new Reader/Writer whose buffer has the default size" (bufio) — that default is 4096 bytes (defaultBufSize in src/bufio/bufio.go). For bulk streaming this is often too small; reading a multi-MB file 4 KiB at a time is still thousands of syscalls.

Size the buffer to the workload with NewReaderSize/NewWriterSize, which return a reader/writer "whose buffer has at least the specified size" (bufio). A 64 KiB–1 MiB buffer for large sequential transfers cuts syscall count by orders of magnitude; tiny interactive protocols want the small default. Don't guess — benchmark the throughput (route to go-perf-benchmarking-statistics). Note both NewReaderSize and NewWriterSize reuse the argument if it is already a bufio reader/writer of sufficient size, so double-wrapping is cheap, not doubly buffered.

The same applies to a net.Conn: a chatty protocol that writes a header, then a body, then a trailer as three conn.Write calls is three packets / three syscalls. Wrap once and flush per logical message:

bw := bufio.NewWriterSize(conn, 16*1024)
writeHeader(bw)
writeBody(bw)
writeTrailer(bw)
if err := bw.Flush(); err != nil { // one syscall, often one segment
	return err
}

For a stream that needs both directions, bufio.NewReadWriter(r, w) pairs a *Reader and *Writer into one value — but it is just a struct holding the two; you still Flush the writer half yourself.

4. bufio.Scanner and the 64 KB Token Limit

Scanner "provides a convenient interface for reading data such as a file of newline-delimited lines of text … the default split function breaks the input into lines" (bufio). The trap: a single token (e.g. one very long line, or a JSON blob with bufio.ScanLines) cannot exceed the buffer cap. The default cap is MaxScanTokenSize, documented as = 64 * 1024 (bufio).

When a token is too big, scanning stops unrecoverably — "Scanning stops unrecoverably at EOF, the first I/O error, or a token too large to fit in the Scanner.Buffer" (bufio). Scan returns false and Err() returns ErrTooLong — errors.New("bufio.Scanner: token too long") (bufio). It does not panic, so a loop that ignores Err() silently processes a truncated prefix of the input — a correctness bug, not just a slow one. Always check scanner.Err() after the loop.

Raise the limit with Scanner.Buffer, which "sets the initial buffer to use when scanning and the maximum size of buffer that may be allocated during scanning" (bufio):

sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) // start 64KiB, grow to 4MiB max
for sc.Scan() {
	process(sc.Bytes()) // Bytes() is only valid until the next Scan
}
if err := sc.Err(); err != nil { // catches ErrTooLong and I/O errors
	return err
}

For genuinely unbounded lines, don't pick a huge cap — use bufio.Reader.ReadString('\n') / ReadLine and stream, so one pathological line can't force a multi-MB allocation. sc.Bytes() aliases the internal buffer; copy it (append([]byte(nil), sc.Bytes()...) or sc.Text()) if you retain it past the next Scan.

5. io.Copy and the ReaderFrom/WriterTo Fast Paths

To move all of one stream into another, reach for io.Copy, not a hand-rolled Read/Write loop. Beyond being shorter, it auto-selects a fast path: "If src implements WriterTo, the copy is implemented by calling src.WriteTo(dst). Otherwise, if dst implements ReaderFrom, the copy is implemented by calling dst.ReadFrom(src)" (io). Those interfaces let a type move data "without … a buffer" of your own — WriterTo.WriteTo "writes data to w until there's no more data to write," ReaderFrom.ReadFrom "reads data from r until EOF" (io). This is the hook that lets file→socket copies drop into the kernel sendfile/splice path (depth: go-perf-zerocopy-and-syscalls).

When neither interface applies, io.Copy allocates one internal 32 KiB buffer (src/io/io.go) and loops. If you copy repeatedly, reuse one buffer with io.CopyBuffer, which "is identical to Copy except that it stages through the provided buffer (if one is required) rather than allocating a temporary one" (io). Two caveats from the same doc:

• It panics on a zero-length buffer: "if buf is nil, one is allocated; otherwise if it has zero length, CopyBuffer panics" — pass make([]byte, 32*1024), never []byte{} (io).
• The buffer is ignored when a fast path exists: "If either src implements WriterTo or dst implements ReaderFrom, buf will not be used" (io) — so don't wrap an *os.File in bufio just to feed io.Copy; the wrapper hides the ReaderFrom/WriterTo method set and can disable the zero-copy fast path.

// Reuse one buffer across many copies; check the error.
buf := make([]byte, 64*1024)
for _, job := range jobs {
	if _, err := io.CopyBuffer(job.dst, job.src, buf); err != nil {
		return err
	}
}

6. Whole-File vs Streaming — Don't Load a Huge File Into RAM

For a small, bounded file, os.ReadFile is correct and clear: it "reads the named file and returns the contents … Because ReadFile reads the whole file, it does not treat an EOF from Read as an error" (os). But it returns the entire file as one []byte — on a multi-GB file or untrusted/unbounded input that is an OOM, not an optimization.

The same trap applies to io.ReadAll, which "reads from r until an error or EOF" (io) — fine for a known-small body, a memory bomb on an unbounded reader (cap it with io.LimitReader first). For large files, stream: wrap in bufio.NewReaderSize and process incrementally, or io.Copy straight to the destination so peak memory stays at one buffer regardless of file size. To write a whole small buffer at once use os.WriteFile (note: it "requires multiple system calls … a failure mid-operation can leave the file in a partially written state" (os)).

Rule of thumb: bounded and small → os.ReadFile/os.WriteFile; large or unbounded → buffered stream / io.Copy.

7. Don't

• Don't issue small unbuffered writes/reads on an *os.File or net.Conn — one syscall each; wrap in bufio (bufio).
• Don't forget bufio.Writer.Flush — un-flushed bytes are silently lost, producing a truncated file with no error (bufio).
• Don't ignore the Flush error — bufio.Writer defers write errors to Flush, so a failed disk write surfaces only there (bufio).
• Don't ignore Scanner.Err() — on a token over 64 KB, Scan returns false with ErrTooLong and the rest of the input is silently dropped; raise the cap with Scanner.Buffer (bufio).
• Don't pass a zero-length buffer to io.CopyBuffer — it panics; pass nil or a sized buffer (io).
• Don't bufio-wrap a file solely to feed io.Copy — it hides ReaderFrom/WriterTo and can disable the kernel sendfile fast path (io).
• Don't os.ReadFile/io.ReadAll an unbounded or huge input — stream it instead, or cap with io.LimitReader (os).
• Don't allocate your own copy buffer when io.Copy would — let it pick the fast path; reuse one buffer via io.CopyBuffer only across repeated copies (io).

8. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-zerocopy-and-syscalls — kernel sendfile/splice (the ReaderFrom/WriterTo payoff), net.Buffers writev, mmap, batching syscalls.
• go-perf-strings-bytes-zerocopy — building strings/bytes without copies (strings.Builder, strconv.Append*, unsafe.String/Slice).
• go-perf-sync-pool — pooling and reusing the copy/scan buffers this skill sizes.
• go-perf-benchmarking-statistics — proving a buffer-size change is a real win, not noise.
• go-perf-methodology — the "should I optimize this I/O at all" judgment.

Parent / sibling marketplace (go-* correctness):

• go-performance — the measure-first overview this skill deepens.
• go-strings-bytes-runes — string/byte/rune correctness behind the perf moves.

9. Reference Files

Buffered-I/O anti-patterns in LLM-generated Go, each with wrong/right contrast and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml