Core Dumps

Core dumps

Contract

Inputs

1. Core file or crash record (required): a core file, a coredumpctl entry, or a /cores/core.<PID> file on macOS.
2. Binary (required): the exact executable that crashed, ideally the unstripped build.
3. Build ID or debug package access (optional): needed when the binary is stripped.

Procedure

1. Enable cores on Linux.

   ulimit -c unlimited                 # this shell only
   ulimit -c                           # confirm
   cat /proc/self/limits               # per-process view
   

   Persist for all users in /etc/security/limits.conf:

   *   soft   core   unlimited
   *   hard   core   unlimited
   

   Control where cores land:

   cat /proc/sys/kernel/core_pattern
   sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t
   

   %e is the executable name, %p the PID, %t the timestamp. If the pattern starts with |, a pipe handler such as systemd-coredump or apport owns core collection; use step 2 instead of looking for files. Done when: ulimit -c reports unlimited and the pattern names a writable path or a known pipe handler.

2. Use coredumpctl when systemd collects cores.

   coredumpctl list                    # recorded crashes
   coredumpctl list myapp              # crashes of one executable
   coredumpctl info                    # details of the latest
   coredumpctl gdb                     # open the latest in GDB
   coredumpctl gdb 12345               # open a specific PID
   coredumpctl dump 12345 -o myapp.core   # export the core file
   

   Cores live in /var/lib/systemd/coredump/. Done when: the crash is listed and coredumpctl gdb opens it.

3. Enable cores on macOS.

   ulimit -c unlimited
   ls /cores/                          # cores land as /cores/core.<PID>
   

   Crash Reporter also writes .crash and .ips logs under ~/Library/Logs/DiagnosticReports/. Done when: a core or crash report exists for the failed run.

4. Analyze the core with GDB.

   gdb ./prog core.12345
   gdb ./prog-with-symbols core.12345  # use the unstripped build when stripped
   

   (gdb) bt                            # call stack
   (gdb) bt full                       # stack plus locals
   (gdb) info registers                # CPU state at the fault
   (gdb) frame 2                       # jump to a frame
   (gdb) info locals
   (gdb) print ptr
   (gdb) thread apply all bt full      # every thread
   (gdb) print $_siginfo               # signal details on Linux
   

   Done when: the crashing frame, the signal, and the faulting address are named.

5. Analyze the core with LLDB.

   lldb ./prog -c core.12345
   

   (lldb) target create ./prog --core core.12345   # equivalent, inside LLDB
   (lldb) bt
   (lldb) thread backtrace all
   (lldb) frame select 2
   (lldb) frame variable
   (lldb) register read
   

   Done when: the crashing frame and faulting access are named.

6. Resolve missing symbols with debuginfod. debuginfod maps build IDs to DWARF data over HTTP.

   export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"
   gdb ./prog core                     # GDB fetches symbols automatically
   debuginfod-find debuginfo <build-id>
   debuginfod-find source <build-id> /path/to/file.c
   

   Done when: symbols resolve, or the build ID is recorded for manual lookup.

7. Resolve symbols manually when debuginfod cannot help.

   readelf -n ./prog | grep 'Build ID'
   

   Install the matching debug package (prog-dbgsym or prog-dbg on Debian, prog-debuginfo on Fedora/RHEL), then point GDB at it:

   (gdb) set debug-file-directory /usr/lib/debug
   

   Done when: GDB loads the matching debug file and bt shows function names.

8. Strip for distribution while keeping symbols. Build with symbols, split them out, ship the stripped binary.

   objcopy --only-keep-debug prog prog.debug
   objcopy --strip-debug prog prog.stripped
   objcopy --add-gnu-debuglink=prog.debug prog.stripped
   

   eu-strip -f prog.debug prog does the split in one step. Keep prog.debug in a symbol store indexed by build ID. Done when: the stripped binary resolves symbols through its .gnu_debuglink.

9. Triage without an interactive session.

   gdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt
   gdb -batch -ex 'info registers' ./prog core
   file core                           # signal, PID, architecture
   

   Done when: crash.txt holds the backtrace and register state.

For the core-pattern token table, the public debuginfod server list, and the full command set, see references/cheatsheet.md.

Failure and recovery

• No core file exists: check ulimit -c, the kernel.core_pattern target, and whether a pipe handler intercepted the dump. Re-run the failing program after fixing the limit.
• coredumpctl list is empty: cores may go to files instead; check core_pattern and the filesystem it names.
• GDB reports no debugging symbols found: fetch them through debuginfod (step 6) or install the debug package (step 7). A stripped binary still yields addresses and a usable bt skeleton.
• The core does not match the binary: GDB warns about mismatched build IDs. Locate the exact binary by build ID; do not trust a same-named rebuild.
• Corrupted stack: bt stops early or shows ?? frames. Read info registers and walk the stack pointer manually with x/.

Output

A crash verdict naming the signal, the faulting frame and address, and the register state, plus the symbol-resolution path used (unstripped binary, debug package, or debuginfod).