Keep a Headless Raspberry Pi Alive with the Built‑in Watchdog and systemd

Problem: A Headless Pi Can Hang Without Warning

In many IoT or kiosk deployments the Raspberry Pi runs without a display. If the operating system freezes—whether from a kernel panic, a hung daemon, or a stuck network stack—you have no visual cue to recover. Rebooting by pulling power is not always feasible, especially in a remote or locked enclosure. The question becomes: how can we guarantee the Pi will reboot itself when it stops responding?

Thesis: Use the Broadcom Watchdog Timer with systemd

All Raspberry Pi models include a hardware watchdog timer controlled by the Broadcom SoC. Linux exposes it as /dev/watchdog via the bcm2835_wdt kernel module. When enabled, the timer forces a hard reset if it is not “kicked” (written to) within its timeout—typically 15 s on the classic BCM2835‑series chips. By letting systemd ping this device automatically, you get a zero‑code, zero‑daemon solution that works even when userspace has frozen.

Why systemd Works Here

systemd can act as the watchdog keeper without any extra process. Setting RuntimeWatchdogSec=10 in /etc/systemd/system.conf tells systemd to open /dev/watchdog once and write a heartbeat every ten seconds. If systemd itself hangs, the kernel will not be able to keep the timer alive and the SoC will reset the board. This covers most userspace freezes and even kernel panics, because the counter lives in the SoC, not in Linux.

Step‑by‑Step Setup

1. Verify the watchdog device exists. On the Pi, run as root:

   ls -l /dev/watchdog
   # Expected: brw-rw---- 1 root root 10, 229 /dev/watchdog
   
   dmesg | grep -i watchdog
   # Look for lines like: "bcm2835 watchdog: enabled, timeout 15s"
   

   If the module is not loaded, install raspberrypi-kernel or load it manually with modprobe bcm2835_wdt.

2. Configure systemd. Edit /etc/systemd/system.conf (or create a drop‑in) and add:

   RuntimeWatchdogSec=10
   

   This value must be less than the hardware timeout (15 s). Save and reboot.

3. Confirm systemd armed the watchdog. After reboot, run:

   systemctl show | grep -i watchdog
   # Expect output like: WatchdogSec=10
   

   You can also check journalctl -b | grep watchdog for “watchdog armed” entries.

4. Test the reboot logic. On a non‑production Pi, simulate a hang by stopping systemd’s timer:

   sudo systemctl stop systemd-journald
   # or use a quick kernel panic: echo c > /proc/sysrq-trigger
   

   The board should reset within ~10 s. After reboot, inspect the journal:

   journalctl -b -1 -e
   # Look for a line: "watchdog: reset" or "kernel: watchdog: reset"
   

   Verify that the reboot was caused by the watchdog, not a power cycle.

Trade‑offs and Practical Limits

• Short Timeout. The hardware timer is limited to ~15 s. If you need a longer “watchdog window”, you must keep RuntimeWatchdogSec below that threshold; larger values are silently ignored by the kernel.
• Hard Reset Risks. The watchdog performs a cold reset. Unsynced filesystems can become corrupted. Mitigate by using a journaling filesystem (e.g., ext4 with data=ordered), a read‑only root, or booting from USB/NVMe where the SD card is not the main storage.
• Masking Symptoms. A watchdog will reboot a hung system but does not diagnose the root cause. Log boot counts and enable persistent journaling so you can investigate recurring resets.
• Hardware Variations. Newer boards (Pi 5, RP1) use a different SoC and may expose the watchdog under a different device node or have a different timeout. Always run dmesg | grep watchdog on the target hardware.

When You Need More Intelligence

If you want the watchdog to consider system health beyond a simple heartbeat, install the watchdog daemon (available in the Raspbian repositories). It can monitor CPU load, memory usage, or network latency, and only stop kicking when a threshold is exceeded. This adds a layer of graceful degradation but requires the daemon to be running; a kernel panic will still trigger the hardware reset.

Actionable Closing

For most headless deployments, simply enabling RuntimeWatchdogSec=10 in system.conf gives you a reliable self‑reboot mechanism that survives userspace hangs and kernel panics. Verify the device exists, test the reboot path on a spare unit, and pair the watchdog with journaling or a read‑only root to avoid data loss. If you need to detect subtle degradation before a hard reset, supplement with the watchdog daemon and keep an eye on the log.