Designing Around Beats Graceful Shutdown: close(), Spool Flush, and the SIGKILL Gap

If you run Filebeat, Metricbeat, or any other Beat under systemd, Kubernetes, or a container orchestrator, the moment that decides whether you lose log lines is not startup — it is shutdown. The Beat's close() path flushes pending events and persists registry state, but only if the process gets a catchable signal and enough time. This note lays out the requirements, the smallest design that protects your data, and the failure modes that would justify something heavier.

Requirements

A shutdown design for a Beat has to satisfy four constraints:

• No silent event loss. Events already read from a file or collected from a metric source must either reach the output or be recoverable on restart.
• Bounded shutdown time. Orchestrators will not wait forever; the design must finish inside the grace period the platform grants.
• Idempotent state. The registry (for Filebeat, the file that records harvester offsets) must be written atomically enough that a crash mid-write does not corrupt it or cause massive re-reading.
• Observable outcome. You must be able to tell, after the fact, whether shutdown was clean.

The smallest suitable design

Beats already implement most of this. On SIGTERM or SIGINT, the Beat's signal handler invokes the internal close() sequence: it stops inputs and harvesters, lets the in-memory pipeline drain into the output, persists registry state, and releases file handles and sockets. If the output (Elasticsearch, Logstash, Kafka) is blocked, pending events are held and, where configured, written through the disk-backed queue (the spool) before exit. This behavior is stable across the 7.x and 8.x lines, with one notable difference: shutdown timeout handling was tightened around 8.5, and older versions could block longer while waiting for a stuck output.

The smallest design, then, is not code — it is configuration of the environment around the Beat:

1. Run the Beat as a supervised service (systemd or a container), never with nohup and a manual kill.
2. Give the supervisor a grace period that exceeds your worst-case drain time.
3. Enable the disk queue if the output can be unavailable for longer than you can tolerate in-memory buffering.
4. Alert on unclean shutdowns rather than assuming them away.

systemd example

On a Linux host with the packaged Beat, drop in an override with systemctl edit filebeat (requires root):

[Service]
TimeoutStopSec=60
KillSignal=SIGTERM

TimeoutStopSec is the window systemd waits after sending SIGTERM before escalating to SIGKILL. Set it from measurement, not guesswork: watch how long a drain takes when the output is slow. SIGKILL is uncatchable — the close() path never runs, and anything still in the in-memory queue is gone.

Kubernetes equivalent

For a Beat DaemonSet, the equivalent knob is terminationGracePeriodSeconds on the pod spec (default 30). If your drain can exceed that, raise it. A preStop hook that sleeps briefly is sometimes suggested, but it does not help the Beat flush — it only delays the SIGTERM. Prefer sizing the grace period correctly.

Trust and data boundaries

Three boundaries matter:

• Source to harvester. Filebeat tracks read offsets in the registry file (data/registry by default). Offsets are persisted during shutdown; if shutdown is killed, the registry may lag, and the Beat will re-read some lines on restart. Downstream, this appears as duplicates, not loss — your consumers should tolerate them.
• Harvester to queue. The in-memory queue is volatile. The disk queue (spool) survives restarts but not a mid-write SIGKILL in all cases; treat it as loss-reduction, not a guarantee.
• Queue to output. Events acknowledged by the output are safe. Events sent but unacknowledged when the process dies will be re-sent, another duplicate source.

The practical consequence: exactly-once delivery is not a property this design provides. Design consumers to deduplicate on a stable document ID if duplicates are harmful.

Operational checks

Verify the shutdown path before you depend on it. Run the Beat in the foreground with -e (logs to stderr) as the service user, then from another shell send SIGTERM to the process:

kill -TERM <beat-pid>

Expected log lines indicate harvesters stopping, the pipeline draining, and the registrar checkpointing state. After exit, confirm the registry file's modification time matches the shutdown moment and, if the disk queue is enabled, inspect the queue directory for retained events. Do this once with the output reachable and once with it blocked (for example, stop Elasticsearch temporarily) to see both drain paths. Do not test with kill -9 and then conclude anything about graceful behavior.

Failure modes

• SIGKILL escalation. Grace period too short, or the kernel OOM killer. In-memory events lost; registry possibly stale. Symptom: duplicates and gaps after restart.
• Blocked output, no disk queue. The Beat waits on the output until the supervisor kills it. On 8.5 and later the shutdown timeout bounds this; older versions can hang longer.
• Full spool. The disk queue is finite. A long output outage fills it, and back-pressure propagates to harvesters — new lines stop being read rather than being lost, which is usually the right trade.
• Registry on a network filesystem. Slow or non-atomic writes during close can corrupt state. Keep the data directory on local disk.

When this design is not enough

Reconsider if any of these hold: your platform's maximum grace period is shorter than your measured drain time and cannot be changed; you cannot tolerate any duplicates downstream (then you need a queuing intermediary like Kafka or Logstash with persistent queues, shifting the durability boundary off the host); or the Beat runs on ephemeral nodes that can vanish without any signal at all (spot instances, hard power loss) — in that case treat the Beat's local state as disposable and rely on source-side retention, such as reading from a log shipper's own durable buffer rather than files only the node holds.

For the common case — a supervised Beat, a sane grace period, a disk queue, and a duplicate-tolerant consumer — the built-in close() path is the whole design. Your job is to give it a signal and enough seconds.