Hunt a memory leak with heap dumps
A leak is memory retained by something still reachable. Prove it by diffing two heap snapshots taken minutes apart under steady load: the type that grows is the retainer. Raising the limit only delays the OOM.
Procedure
Confirm it is a leak rather than a cache warming up. Sample RSS at constant traffic: while true; do ps -o rss= -p $PID; sleep 60; done Monotonic growth past warmup with no plateau is a leak; a plateau is a bounded cache.
Capture two heap profiles five minutes apart under the same load: go tool pprof -output=h1.pb.gz http://127.0.0.1:6060/debug/pprof/heap sleep 300 go tool pprof -output=h2.pb.gz http://127.0.0.1:6060/debug/pprof/heap
Diff them — only the delta matters: go tool pprof -base=h1.pb.gz -inuse_space -top -nodecount=20 h2.pb.gz The top frame is the allocator holding the growth.
For a JVM, take
jmap -dump:live,format=b,file=h1.hprof <pid>twice, open both in Eclipse MAT, run Leak Suspects, and compare dominator trees.Classify the retainer by pattern: an unbounded map keyed by user id with no TTL; goroutines blocked on a channel that never drains; a
[]bytebuffer grown and never trimmed; listener/observer registrations never removed; acontextstored in a long-lived struct.Falsify alternatives with a targeted experiment before fixing: hit the suspect endpoint 1000 times, recapture, and show the retainer's
inuse_spacegrew roughly linearly with calls. Growth tied to a specific path excludes the other suspects.Fix, then prove the plateau: run 30 minutes at load and require RSS to stay within 5% across the last ten samples.
Pitfalls
- Reading
alloc_spaceand calling it a leak: it counts all allocation ever and grows by definition. Onlyinuse_spacemeasures retention. - A "leak" that is really the GC not having run — force
runtime.GC()(Go) orSystem.gc()(JVM) and re-readinusebefore concluding. - Fixing the symptom by shrinking payloads to slow the growth while the unbounded map remains.
- One snapshot at one instant: without a diff you cannot separate the working set from the leak.
Verification
go tool pprof -base=h1.pb.gz -inuse_space h2.pb.gz <<< 'top5'
ps -o rss= -p $PID # last 10 one-minute samples within 5% after the fix
Report: the retainer type, its growth rate in MB/hour before the fix, and the RSS plateau after with the sampling window that proves it.