Core dumps
A core dump is the process frozen at the instant it died: registers, stack, and heap preserved. It is the only witness to a crash you cannot reproduce. Wasted without symbols it is a wall of hex, so the whole job is turning that snapshot back into named functions, arguments, and the one bad pointer.
Method
- Make sure cores are actually written. Check
ulimit -c(often0, which silently discards them) and setulimit -c unlimited. Find where they land viacat /proc/sys/kernel/core_pattern; systemd hosts route them tocoredumpctl, socoredumpctl listandcoredumpctl dumppull the file. No core means no post-mortem, so fix this before the next crash. - Match the binary and symbols to the exact build. Load the dump with
the identical executable that produced it:
gdb ./app core.1234. Mismatch by one commit and every address resolves to the wrong function. Install the matching-dbg/debuginfopackage or point at your separated.debugfile sogdbcan map addresses to names. - Read the backtrace first.
btprints the call stack at death;bt fulladds local variables per frame. The top frame is where the signal hit (SIGSEGV,SIGABRT), and the frames below are how it got there.info registersshows the faulting address in the offending register. - Walk to the frame you own and inspect state.
frame Nselects a frame,print varandp *ptrdump values,x/16xw $spexamines memory around the stack pointer. ASIGSEGVon a read of0x0means a null deref: find which variable holds that null and which frame set it. - Read the signal as a clue, not just a fact.
SIGSEGVis a bad address,SIGABRTis usually an assertion orabort()from a detected invariant,SIGBUSis misalignment or a truncated mmap,SIGFPEis a divide by zero. The signal narrows the class of bug before you read code. - Cross-check with the allocator and sanitizer story. A freed pointer
reused (
0xdeadbeef-style poison, or glibc'sfree(): invalid pointer) points at use-after-free or double-free. Rebuild under AddressSanitizer to catch it live once you know roughly where, since a dump shows the wreck but not always the moment of corruption.
Checks
- Does every frame in
btresolve to a function name, not a bare address? - Does the faulting address in the registers match a variable you can name?
- Does the signal type agree with the code path the backtrace points to?
Boundaries
Cores from stripped release binaries, JIT-compiled runtimes, or heavily
optimized code give partial or misleading stacks; inlining erases frames and
optimized-out locals read as garbage. Managed runtimes (JVM, .NET) have their
own dump tools (jcmd, dotnet-dump) that beat gdb on those stacks.