Capturing Kernel Panics with kdump: Setup and Validation Guide

The Problem: The Silent Crash

When a Linux kernel panics, the system typically freezes or reboots instantly. Without a memory dump, you are left with nothing but a hope that the logs reached the disk before the crash. To diagnose the root cause—such as a driver bug or memory corruption—you need a vmcore: a snapshot of the system's RAM at the exact moment of failure.

The solution is kdump. It uses kexec to boot into a secondary, minimal "capture kernel" already residing in a reserved slice of RAM. When the primary kernel crashes, it jumps to this capture kernel, which saves the memory state to disk without relying on the corrupted state of the crashed system.

Prerequisites and Constraints

• Root Privileges: All configuration and testing require sudo or root access.
• Reserved Memory: You must sacrifice a small portion of system RAM (typically 128MB to 512MB) to house the capture kernel. This memory is unavailable to your applications.
• Matching Debug Symbols: To actually analyze the resulting dump, you will need the vmlinux file (debug symbols) matching your exact kernel version.
• Secure Boot: If UEFI Secure Boot is enabled, the capture kernel must be signed, or the system may block the kexec jump.

Step 1: Reserving Memory via GRUB

The capture kernel cannot be loaded into general-purpose RAM because the crash itself might have corrupted that memory. You must reserve a dedicated area at boot time using the crashkernel parameter.

1. Edit your GRUB configuration (usually /etc/default/grub).
2. Add crashkernel=256M to the GRUB_CMDLINE_LINUX_DEFAULT line.
   Note: On some distributions, crashkernel=auto is supported, but explicit values (e.g., 256M or 512M) are more reliable for ensuring the capture kernel has enough headroom to avoid OOM (Out of Memory) during the dump.
3. Update the GRUB config:

   # Ubuntu/Debian
   sudo update-grub
   
   # RHEL/CentOS/Fedora
   sudo grub2-mkconfig -o /boot/grub2/grub.cfg

4. Reboot the system to apply the memory reservation.

Step 2: Installing and Configuring the Capture Service

Depending on your distribution, the toolset differs, but the logic remains the same: load the capture kernel into the reserved space.

RHEL / CentOS / Fedora

sudo yum install kexec-tools
sudo systemctl enable --now kdump

Ubuntu / Debian

sudo apt-get install kdump-tools
# During installation, a prompt will ask if you want to enable kdump. Select Yes.

By default, dumps are stored in /var/crash. If you are running a virtual machine or a cloud instance, consider configuring a remote NFS or SSH target in /etc/kdump.conf to prevent the dump from filling the root partition and causing a secondary failure.

Step 3: Verifying the Loaded State

Before triggering a crash, verify that the capture kernel is actually resident in memory. If it isn't, the system will simply reboot on panic without saving any data.

Run the following check as root:

cat /sys/kernel/kexec_crash_loaded

Expected Result: The command should return 1. If it returns 0, check systemctl status kdump for errors regarding insufficient memory reservation.

Step 4: End-to-End Validation (The Forced Panic)

Warning: This operation will crash your system immediately. Do not perform this on a production server. Ensure all data is saved.

To simulate a kernel panic, use the Magic SysRq trigger:

1. Enable SysRq (if not already enabled):

   sudo sysctl -w kernel.sysrq=1

2. Trigger the crash:

   echo c | sudo tee /proc/sysrq-trigger

The system will freeze and then automatically reboot into the capture kernel. You will see a console message indicating that the vmcore is being saved. Once the system reboots back to the normal OS, check for the dump file:

ls -lh /var/crash/

You should see a directory timestamped with the crash time containing a vmcore file.

Diagnostic Decision: Analysis vs. Extraction

Goal	Tool	Requirement
Quick Log Check	makedumpfile	Only the vmcore file. Extracts dmesg fragments.
Full Root Cause	crash utility	vmcore + matching vmlinux (debug symbols).

Recovery and Troubleshooting

• Capture Kernel OOM: If the system reboots without a dump, the crashkernel reservation may be too small for your specific kernel version. Increase it to 512M and reboot.
• Disk Full: If /var/crash is on a small partition, the dump may be truncated. Use a dedicated disk or network mount.
• Rollback: To disable kdump and reclaim RAM, remove the crashkernel parameter from GRUB, update GRUB, and reboot.