Diagnosing Linux 'Task Blocked for More Than 120 Seconds' Warnings

Your kernel log shows INFO: task postgres:1234 blocked for more than 120 seconds, followed by a stack trace, and the machine is still running — mostly. The useful takeaway: this message comes from the kernel's hung-task watchdog (khungtaskd), and it is a detector, not a diagnosis. The stack trace in the same log entry usually tells you which subsystem the task is stuck in, and that points you at disk, network filesystem, or driver causes. Killing the named process almost never works, because it is in uninterruptible sleep.

What the warning actually means

A kernel thread named khungtaskd periodically scans all tasks in D state (uninterruptible sleep, typically waiting on I/O). Any task that stays in D state longer than kernel.hung_task_timeout_secs (default 120 seconds) gets logged with its stack trace. D-state tasks cannot be interrupted by signals — including SIGKILL — because the kernel assumes they are mid-operation on hardware or a filesystem.

Occasional warnings during a known storage hiccup are noise. Repeated warnings, warnings naming the same device, or warnings that coincide with application stalls are a real problem worth working through.

Matching the stack trace to a cause

Ordered checks

1. Capture the stuck task's state before anything recovers. As root, run cat /proc/<pid>/stack and cat /proc/<pid>/status (check the State: line). The kernel log already has one trace, but a second sample minutes later shows whether the task is making progress or truly wedged. If /proc/<pid>/stack itself hangs, that is also informative — the task is deep in the kernel.
2. Review dmesg around the timestamp. Run dmesg -T | less and look for I/O errors, SCSI device resets (ata1: link is slow to respond, blk_update_request: I/O error), or filesystem errors (EXT4-fs error, XFS: metadata I/O error) just before the hung-task entry. These almost always share a root cause.
3. Check device latency. Run iostat -x 5 3 (from the sysstat package, works as any user). A device pinned at 100% %util with await in the hundreds or thousands of milliseconds is your suspect. On virtual machines, high guest-side await with healthy-looking virtual disks often means the hypervisor's storage is slow — guest diagnostics cannot see that layer, so check the host's storage metrics too.
4. If NFS is involved, test the server directly. From the client: showmount -e <server> or a simple ls on the mount from a shell (note: that ls may itself hang in D state — run it with a timeout from another session). Check mount | grep nfs for hard vs soft options; hard mounts retry forever and produce exactly this symptom when the server disappears.
5. Check memory pressure if the trace shows reclaim. Read /proc/pressure/memory (requires the PSI feature, present since kernel 4.20) and vmstat 5 3 for swap I/O.

Fixes tied to what you find

NFS stall: restore the server or network path first. If the mount must survive server outages, remount with softer semantics, e.g. mount -o remount,soft,timeo=50,retrans=2 <mountpoint> as root — but understand that soft can return I/O errors to applications and risk silent data corruption on writes, so prefer fixing the server. A wedged hard mount sometimes only clears with umount -f or a reboot.

Failing disk: check smartctl -a /dev/sdX (smartmontools, root) for reallocated or pending sectors, schedule replacement, and run a filesystem check during a maintenance window. Do not keep trusting a device that logs I/O errors.

Single wedged process: kill -9 will appear to do nothing until the underlying I/O completes or times out. That is expected behavior, not a failed command. If the I/O never completes, only clearing the underlying stall (or rebooting) frees the task.

Do not "fix" it by tuning the watchdog. Raising kernel.hung_task_timeout_secs only silences the alarm. Setting kernel.hung_task_panic=1 converts hangs into panics — useful on a test host paired with kdump so you capture a vmcore for analysis, dangerous on a production box without that evidence pipeline. Check current values with sysctl kernel.hung_task_timeout_secs kernel.hung_task_panic.

Validating on a test VM

You can reproduce the detector safely: mount a FUSE filesystem whose daemon you then stop, or use a device-mapper delay/error target to stall block I/O, then touch the mount. After the configured timeout, dmesg should show the hung-task warning naming your test process. This confirms your logging pipeline captures these events before you rely on it in production. Do this only on a disposable VM.

When to escalate

• Hangs recur on the same device after disk replacement — suspect the controller, cabling, or hypervisor storage layer.
• Stack traces are identical across reboots on a fully patched kernel — that pattern suggests a real kernel or driver bug. File a report with your distro or the subsystem maintainer, including the full hung-task traces and, if you configured hung_task_panic with kdump, the vmcore.
• Warnings correlate with application-visible outages you cannot tie to storage — collect /proc/<pid>/stack samples and iostat history over several days, then involve your platform or storage vendor.

After any fix, watch dmesg and iostat -x under representative load for several days. Silence from khungtaskd plus normalized await times is your confirmation; a quiet log during idle hours proves nothing.