kernel-debugging

name	kernel-debugging
description	Linux kernel debugging skill for kgdb, kdb, ftrace, kprobes, and crash analysis. Use when setting up kgdb over serial, using ftrace and trace-cmd, inserting kprobes, enabling dynamic debug, or analyzing kdump with crash. Activates on queries about kgdb, kdb, ftrace, kprobes, dyndbg, kdump, or kernel dmesg levels.

Kernel Debugging

Purpose

Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and trace-cmd, kprobes and kretprobes, dynamic debug (dyndbg), crash dump analysis with crash and makedumpfile, dmesg log levels, and /proc/sys runtime tuning for debugging.

When to Use

• Kernel panic or oops with unclear stack trace
• Live debugging a kernel module or driver with kgdb
• Tracing function calls without recompiling (ftrace, kprobes)
• Enabling verbose driver logging with dyndbg
• Analyzing production crash dumps from kdump
• Tuning kernel runtime parameters for debug sessions

Workflow

1. dmesg and log levels

# Kernel ring buffer
dmesg -T -l err,warn
dmesg -w   # follow live

# Log level (0=emerg .. 7=debug)
cat /proc/sys/kernel/printk
# current  default  minimum  boot-default
# 4        4        1        7

# Temporarily increase verbosity
echo 8 > /proc/sys/kernel/printk

Kernel printk levels: KERN_EMERG through KERN_DEBUG. Driver dev_dbg() requires dyndbg or DEBUG define.

2. kgdb over serial

Kernel config: CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE.

# Boot parameter (ttyS0, 115200)
kgdboc=ttyS0,115200 kgdbwait

# Or at runtime
echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
echo g > /proc/sysrq-trigger   # break into kgdb

GDB host side:

# Connect to remote serial
gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
(gdb) bt
(gdb) list *panic+0x20

USB variant: CONFIG_KGDB_KDB with kgdboc=usb on supported hardware.

3. kdb commands

When CONFIG_KGDB_KDB enabled, kdb provides in-kernel shell:

kdb commands (at SysRq break)
├── bt          — stack backtrace
├── lsmod       — loaded modules
├── md <addr>   — memory display
├── ps          — process list
├── id <cpu>    — CPU registers
├── go          — continue execution
└── cpu <n>     — switch CPU context

# Enter kdb
echo k > /proc/sysrq-trigger

4. ftrace

# Enable function tracer
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on

# Filter to specific function
echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':*probe_*' >> /sys/kernel/debug/tracing/set_ftrace_notrace

# Capture and read
cat /sys/kernel/debug/tracing/trace_pipe

# Disable
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer

# trace-cmd (userspace recorder)
trace-cmd record -p function -l schedule,do_page_fault
trace-cmd report
trace-cmd.dat  # kernelshark GUI

5. kprobes and kretprobes

# Dynamic kprobe via debugfs
echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
cat /sys/kernel/debug/tracing/trace

# kretprobe
echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events

# Cleanup
echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events

In-kernel kprobe (module):

#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = my_pre_handler,
};

static int __init mod_init(void)
{
    return register_kprobe(&kp);
}

6. Dynamic debug (dyndbg)

# Enable all debug messages for a module
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control

# Enable specific file:line
echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control

# Query current settings
grep mydriver /sys/kernel/debug/dynamic_debug/control

Boot-time: dyndbg="module mydriver +p" on kernel command line.

7. kdump and crash analysis

# Setup kdump (crash kernel reserved memory)
# /etc/default/grub: crashkernel=256M
sudo kdump-config show

# After crash, vmcore in /var/crash/
ls /var/crash/*/vmcore

# Analyze with crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore

crash> bt
crash> log
crash> ps
crash> kmem -i
crash> mod

# Compress vmcore
makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo

8. /proc/sys runtime tuning for debugging

# Panic on oops (reproduce in VM)
sysctl kernel.panic_on_oops=1

# Soft lockup detection
sysctl kernel.softlockup_panic=1

# Watchdog
sysctl kernel.nmi_watchdog=1

# Slab debugging
# Boot: slub_debug=P,pagealloc

9. Debugging decision tree

Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit

Common Problems

Symptom	Cause	Fix
kgdb won't connect	Wrong tty/baud	Match kgdboc to serial adapter
ftrace empty trace	tracing_off or nop tracer	echo function > current_tracer; enable tracing_on
kprobe registration fails	Missing CONFIG_KPROBES or inlined function	Use tracepoint or static_key
dyndbg has no effect	CONFIG_DYNAMIC_DEBUG disabled	Rebuild kernel with option
crash can't read vmcore	Wrong vmlinux debug symbols	Install linux-image-$(uname -r)-dbg
SysRq doesn't work	kernel.sysrq disabled	echo 1 > /proc/sys/kernel/sysrq

Related Skills

• skills/low-level-programming/linux-kernel-modules — module development being debugged
• skills/kernel/device-drivers — driver-specific probe/IRQ issues
• skills/kernel/kernel-internals — subsystem knowledge for interpreting traces
• skills/debuggers/gdb — GDB commands shared with kgdb
• skills/profilers/linux-perf — perf for kernel hotspots
• skills/observability/ebpf — non-invasive kernel tracing alternative