k3os and systemd: What is the impact of kubelet's stop timeout on pod termination?

Confirmed behavior: TimeoutStopSec=0 forces immediate SIGKILL

On a fresh k3os installation, the kubelet.service unit file contains TimeoutStopSec=0. When the node is powered down or reset, systemd sends SIGTERM to the kubelet process, waits zero seconds, then sends SIGKILL. This happens regardless of whether pods are still terminating.

What actually happens during shutdown

1. SIGTERM to kubelet: The kubelet receives SIGTERM and begins its termination sequence, which includes respecting each pod's terminationGracePeriodSeconds.
2. Immediate SIGKILL: Because TimeoutStopSec=0, systemd does not wait. Within milliseconds, it sends SIGKILL to the kubelet.
3. Pods left in limbo: Any pod that has not completed its grace period is abruptly terminated, potentially before finishing pre-stop hooks, flushing buffers, or closing connections.

Verification steps

Run these commands on your k3os node to confirm the current state:

# Check current timeout setting
systemctl show kubelet -p TimeoutStopUSec

# Watch kubelet logs during a controlled shutdown
journalctl -u kubelet -f

# From a control plane node, monitor pod termination
kubectl get pods -A --field-selector spec.nodeName=$(hostname) -w

Recommended fix: set a non-zero stop timeout

The practical solution is to set TimeoutStopSec to a value that accommodates the longest-running pod in your cluster. A typical default of 120–300 seconds works for most workloads.

Step 1: Create a systemd drop-in override

sudo systemctl edit kubelet

This opens an editor. Add the following content:

[Service]
TimeoutStopSec=180

Step 2: Verify the override persists

The override is written to /etc/systemd/system/kubelet.service.d/override.conf. However, k3os may overwrite unit files during upgrades. To preserve this setting across upgrades, add it to /oem/override.yaml:

systemd:
  units:
    - name: kubelet.service
      dropin:
        Service:
          TimeoutStopSec: 180

Why this matters

Stateful workloads—databases, message queues, file stores—rely on graceful shutdown to flush data and close connections cleanly. A zero timeout bypasses all of this, increasing the risk of data corruption or inconsistent state. Additionally, pod disruption budgets may be violated if pods are force-killed before their defined disruption policies allow.

One missing diagnostic detail

If you have workloads with unusually long pre-stop hooks (e.g., backup operations before shutdown), you may need to increase the timeout further. Check your pod specs for terminationGracePeriodSeconds and add a safety margin.