Filebeat Harvester Offset Persistence: How Reliable Tailing Survives Restarts

The Problem: Losing Your Place in the Log Stream

You're tailing a busy application log with Filebeat. The process crashes, the host reboots, or you deploy a config change. When Filebeat restarts, does it pick up exactly where it left off—or does it re-send the last few thousand lines, flooding your Elasticsearch cluster with duplicates?

The answer lies in the Harvester's offset persistence mechanism. This isn't magic; it's a deliberate design choice with configurable trade-offs between durability and I/O overhead. Understanding it lets you tune Filebeat for your reliability requirements without guesswork.

How Offset Persistence Works

Each Harvester (the component that reads a single file) maintains an in-memory cursor: the byte offset of the last successfully acknowledged event. Periodically, this cursor is flushed to a state file on disk, typically under /var/lib/filebeat/registry/filebeat/ or data/registry/ depending on your installation.

The state file is a JSON document mapping file identifiers (inode + device) to offset metadata. When Filebeat starts, it reads this registry, matches files by their filesystem identity—not just path—and resumes reading from the stored offset.

Key configuration knobs:

• harvester_flush_interval (default: 1s): How often the in-memory offset is written to disk. Lower values increase durability but raise I/O.
• harvester_limit (default: 0, unlimited): Maximum concurrent harvesters. Prevents runaway file descriptor consumption on hosts with thousands of log files.
• scan_frequency (default: 10s): How often the input scans for new files or rotation events.

Worked Example: Verifying Offset Survival Across Restart

Let's prove the mechanism works with a minimal test. Assume Filebeat 8.x on Linux, running as a service.

# 1. Create a test log and minimal config
mkdir -p /tmp/fb-test/logs /tmp/fb-test/data/registry
cat > /tmp/fb-test/filebeat.yml <<'EOF'
filebeat.inputs:
- type: log
  paths:
    - /tmp/fb-test/logs/app.log
  registry.flush: 1s  # flush offset every second
output.console:
  pretty: true
EOF

# 2. Start Filebeat in background
filebeat -c /tmp/fb-test/filebeat.yml \
  --path.data /tmp/fb-test/data \
  --path.logs /tmp/fb-test/logs \
  -e > /tmp/fb-test/fb.log 2>&1 &
FB_PID=$!
sleep 2

# 3. Write initial lines
echo 'line 1' >> /tmp/fb-test/logs/app.log
echo 'line 2' >> /tmp/fb-test/logs/app.log
sleep 1  # allow flush

# 4. Kill and restart
kill $FB_PID
wait $FB_PID 2>/dev/null
filebeat -c /tmp/fb-test/filebeat.yml \
  --path.data /tmp/fb-test/data \
  --path.logs /tmp/fb-test/logs \
  -e >> /tmp/fb-test/fb.log 2>&1 &
FB_PID=$!
sleep 1

# 5. Append more lines
echo 'line 3' >> /tmp/fb-test/logs/app.log
sleep 1

# 6. Check output: only line 3 should appear after restart
grep -A2 'line 3' /tmp/fb-test/fb.log
# Expected: no 'line 1' or 'line 2' duplicates

# 7. Inspect registry offset
cat /tmp/fb-test/data/registry/filebeat/*/offset.json | jq

After step 6, you should see only "line 3" emitted post-restart. The registry dump in step 7 will show an offset matching the file size after "line 2" (roughly 10 bytes per line including newline).

File Rotation: Inode Tracking Prevents Gaps

When logrotate or your application renames app.log to app.log.1 and creates a new app.log, the Harvester detects the inode change. It closes the old file (already fully read), opens the new inode at offset 0, and continues. The registry stores both the old file's final offset (for cleanup) and the new file's starting position.

This works because Filebeat identifies files by (device, inode) tuples, not paths. A rename preserves the inode; a copy-truncate rotation changes it, and Filebeat handles both.

Trade-offs: Flush Interval vs. Durability

The harvester_flush_interval setting is the primary lever. At the default 1s, a crash loses at most one second of processed events—those acknowledged by the output but not yet persisted in the registry. Setting it to 100ms reduces the window but increases disk writes by 10x. On a host tailing 500 files, that's noticeable I/O.

Conversely, setting it to 30s or disabling periodic flush (relying only on clean shutdown) risks replaying minutes of data after a power loss. For most observability pipelines, 1–5s is a reasonable balance. For audit-critical logs, consider 100–500ms and provision IOPS accordingly.

One caveat: if the registry file itself is deleted or corrupted (e.g., rm -rf /var/lib/filebeat/registry), Filebeat treats all files as new and reads from the beginning. Protect the registry directory with backups or include it in your host backup policy.

Horizontal Scaling: Shared Registry Caveats

Running multiple Filebeat instances on the same files requires a shared registry (e.g., on an NFS mount or via a centralized state store). The documentation supports this, but contention on the registry file can become a bottleneck. Each instance acquires a file lock on write; with many harvesters flushing every second, lock contention adds latency.

If you need horizontal scaling, consider assigning disjoint file sets to each instance (via path globs or exclude_files) rather than sharing a registry. Simpler, more predictable, and avoids the "split brain" risk where two harvesters think they own the same file.

Actionable Checklist

1. Confirm your harvester_flush_interval matches your RPO (recovery point objective). Default 1s is fine for metrics/logs; lower for audit trails.
2. Verify registry persistence: ls -la /var/lib/filebeat/registry/ should show recent offset.json updates.
3. Test rotation: run logrotate -f /etc/logrotate.d/your-app while Filebeat tails, confirm no gaps in output.
4. Back up the registry directory in your host backup strategy—treat it like a database.
5. If scaling horizontally, partition file ownership instead of sharing registry.

Offset persistence is the quiet foundation of Filebeat's exactly-once-ish delivery. Tune the flush interval, protect the registry, and you'll rarely think about it again—until you need to debug a duplicate spike, at which point you'll know exactly where to look.