Diagnosing Duplicate Log Ingestion in Filebeat Registry State

The Problem: Log Duplication Storms

When Filebeat restarts, it should resume reading from the exact byte where it stopped. If you suddenly see thousands of duplicate events in your output (such as Elasticsearch or Logstash) that were already processed days or weeks ago, you are likely experiencing a registry state failure.

The registry is a local database where Filebeat stores the offset—the specific byte position—of every file it tracks. If this state is lost or corrupted, Filebeat treats every known log file as a new source and reads them from the beginning.

Diagnostic Matrix: Identifying the Cause

Symptom	Likely Cause	Diagnostic Indicator
Massive spike in old logs after restart	Registry deletion/corruption	Missing or 0-byte files in path.data/registry
Logs repeat only after a period of inactivity	close_inactive timeout	Filebeat logs showing "Closing file... inactive for Xs"
Registry file grows excessively large	Too many unique small files	High disk I/O on the path.data volume
No logs moving; registry not advancing	Output backpressure	Filebeat logs showing "Publishing events: failed"

Step-by-Step Recovery and Verification

1. Verify Registry Existence and Permissions

Check if the registry directory exists and is writable by the user running the Filebeat process. By default, this is located in the data folder relative to the installation, or defined by the path.data setting in filebeat.yml.

# Run as root or the filebeat user
ls -la /var/lib/filebeat/registry

Expected Result: You should see a directory structure containing state files. If the directory is empty or the permissions are restricted (e.g., owned by root while Filebeat runs as filebeat), the agent cannot save its progress.

2. Analyze the close_inactive Configuration

If logs are duplicating only for files that haven't been updated in a while, check your filebeat.yml for the close_inactive setting. This setting tells Filebeat when to stop tracking a file and remove it from the registry to save memory.

# Example configuration in filebeat.yml
filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
  close_inactive: 5m

Risk: If close_inactive is set too low (e.g., 1m) and a log file is updated infrequently, Filebeat will drop the offset. When the file is finally updated, Filebeat may re-read the entire file from the start depending on the clean_inactive and ignore_older settings.

3. Monitor Offset Advancement

To verify that the registry is actually updating in real-time, use the Filebeat monitoring API (if enabled) or observe the registry files. You can simulate a write to a tracked log and check for activity.

1. Append a test line to a tracked log: echo "registry-test-$(date)" >> /var/log/app.log
2. Check the timestamp of the registry files: ls -l /var/lib/filebeat/registry
3. Verify the output destination receives exactly one event.

Fixes Based on Findings

• If registry files are missing: Restore from a backup of the path.data directory if available. If no backup exists, you must manually truncate the source logs or use a script to delete duplicates at the destination to prevent data pollution.
• If close_inactive is too aggressive: Increase the duration (e.g., from 5m to 24h) to ensure slow-moving logs remain in the registry.
• If disk I/O is too high: Move the path.data directory to a faster disk (SSD) or reduce the number of unique files being tracked by aggregating logs before they reach Filebeat.

Rollback and Safety

Warning: Never manually edit the binary registry files. Doing so will likely corrupt the state, forcing Filebeat to restart from byte 0 for all files.

If you change path.data to move the registry to a new volume, you must manually move the existing registry files to the new location before restarting the service to avoid a full re-ingestion of all logs.

# Example migration
systemctl stop filebeat
mv /var/lib/filebeat/registry /mnt/fast-disk/filebeat/registry
# Update filebeat.yml path.data to /mnt/fast-disk/filebeat
systemctl start filebeat

Escalation Criteria

Contact your infrastructure or storage team if:

• The path.data volume is reporting read-only errors (indicating hardware failure).
• The registry continues to reset despite correct permissions and sufficient disk space.
• The output destination is consistently rejecting events, preventing the registry from advancing (Backpressure).