Kernel Debugging

Kernel debugging

Contract

Inputs

1. Symptom (required): panic, oops, hang, lockup, or wrong behavior.
2. Target access (optional): serial console, root shell, mounted debugfs, or a vmcore file.
3. Kernel build (optional): vmlinux with debug info for the running kernel; mainline 7.2 or LTS 6.18 assumed when not stated.

Procedure

1. Read the ring buffer before anything else.

   dmesg -T -l err,warn              # filter by level
   dmesg -w                          # follow live
   cat /proc/sys/kernel/printk       # current default minimum boot-default
   echo 8 > /proc/sys/kernel/printk  # raise verbosity
   

   Driver dev_dbg() output stays hidden until dynamic debug or a DEBUG define turns it on. Done when: the level is raised and the first fault line is captured.

2. Break into kgdb when the target is alive. Boot with kgdboc=ttyS0,115200 kgdbwait, or attach at runtime and trigger the break from sysrq.

   echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
   echo g > /proc/sysrq-trigger
   

   gdb vmlinux
   (gdb) set serial baud 115200
   (gdb) target remote /dev/ttyUSB0
   (gdb) bt
   

   A USB serial adapter shows up as ttyUSB0 and works the same as a built-in port. Done when: gdb reports a backtrace on the stopped target.

3. Use kdb when no host gdb exists. CONFIG_KGDB_KDB gives an in-kernel shell at the sysrq break (echo k > /proc/sysrq-trigger). Commands: bt backtrace, ps process list, lsmod modules, md <addr> memory, rd registers, id <addr> disassembly, cpu <n> switch CPU, go continue. Done when: the faulting frame is visible without a host debugger.

4. Trace without recompiling: ftrace.

   cd /sys/kernel/debug/tracing
   echo function > current_tracer
   echo schedule > set_ftrace_filter
   echo '*probe*' > set_ftrace_notrace
   echo 1 > tracing_on
   cat trace_pipe
   echo 0 > tracing_on && echo nop > current_tracer
   

   trace-cmd record -p function -l schedule,do_page_fault
   trace-cmd report          # kernelshark reads the same trace.dat
   

   Requires debugfs mounted and root. Done when: the trace shows the calls in question with the filters narrowing it.

5. Probe one suspect function with kprobes.

   echo 'p:myprobe do_sys_open $arg1 $arg2' > /sys/kernel/debug/tracing/kprobe_events
   echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
   cat /sys/kernel/debug/tracing/trace
   echo 'r:myret do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
   # cleanup
   echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events
   

   An in-kernel probe registers a struct kprobe with .symbol_name and a .pre_handler through register_kprobe(). Done when: the probe fires and the fetch arguments read sanely.

6. Turn on driver debug prints with dyndbg.

   echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
   echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control
   grep mydriver /sys/kernel/debug/dynamic_debug/control
   

   Boot time: dyndbg="module mydriver +p" on the kernel command line. Needs CONFIG_DYNAMIC_DEBUG. Done when: the driver's debug lines appear in the log.

7. Analyze the crash dump. Reserve memory with crashkernel=256M, then use the distro tool (kdump-config show on Ubuntu, kdumpctl status on RHEL). After a panic the vmcore lands in /var/crash/.

   crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
   crash> bt
   crash> log
   crash> ps
   crash> kmem -i
   crash> mod
   

   Compress with makedumpfile -c -d 31 /proc/vmcore /tmp/vmcore; the dump level is a number after -d. The vmlinux must carry debug info for the exact running kernel, usually the distro linux-image-*-dbg package. Done when: the crash session yields the panic backtrace.

8. Tune the target for reproduction.

   sysctl kernel.panic_on_oops=1      # stop at the first oops, in a VM
   sysctl kernel.softlockup_panic=1
   sysctl kernel.nmi_watchdog=1
   # boot: slub_debug=P,pagealloc     # poison slab for corruption hunts
   

   Done when: the target halts at the defect instead of limping on.

9. Triage by issue class.

   Panic or oops        -> dmesg, then crash on the vmcore
   Driver logic bug     -> dyndbg, then ftrace function_graph
   Latency regression   -> perf record -g -a, trace-cmd
   Intermittent         -> kprobe the suspect path
   Module crash         -> kgdb, audit module refcounts
   

   Done when: the next tool follows from the class.

Failure and recovery

Output

1. The faulting function or subsystem named with captured evidence.
2. The command transcript used.
3. The next diagnostic step when the cause is still open.