Architecting Filebeat: Direct-to-Elasticsearch vs. Logstash Tiers

The Decision: Filebeat Alone or Logstash?

Adding a processing tier between hosts and Elasticsearch is a common reflex, but it is not always the starting point. The practical question is whether a single Filebeat process on each host can meet delivery, buffering, and normalization requirements, or whether a separate Logstash tier is genuinely necessary. The smallest suitable design is usually Filebeat with a persistent (disk) queue shipping directly to Elasticsearch, relying on Filebeat modules and ingest pipelines for parsing. Index and lifecycle management is handled by Elasticsearch.

Requirements the Design Must Satisfy

Before choosing components, define what the shipper must guarantee. At minimum, a host-level shipper should:

• Read local files with low overhead and track progress so restarts do not re-read from the beginning.
• Buffer events during short outages without blocking inputs.
• Deliver events at least once, accepting possible duplicates rather than silent gaps.
• Keep credentials and unshipped data inside a defined boundary.
• Expose enough metrics and logs to distinguish backpressure from a stuck pipeline.

The Smallest Suitable Design

Filebeat on the source host, configured with a persistent queue and direct Elasticsearch output, satisfies the above for most single-cluster deployments. The persistent queue writes unacknowledged events to local disk and survives restarts and brief output outages. The default memory queue, by contrast, loses in-flight events on crash.

Filebeat modules for common services like system logs or nginx ship with ingest pipelines that parse and normalize inside Elasticsearch, removing the need for a separate parsing tier. Index lifecycle is handled by Elasticsearch using the templates Filebeat sets up.

Example Configuration

A minimal configuration that uses a persistent queue and a least-privilege API key resembles the following. Run Filebeat on the host that owns the log files; the process needs read access to those files and write access to the data and queue directories.

filebeat.inputs:
- type: filestream
  id: app-logs
  paths:
    - /var/log/app/*.log

queue.disk:
  max_size: 10GB
  path: /var/lib/filebeat/queue

output.elasticsearch:
  hosts: ["https://<ES_HOST>:9200"]
  api_key: "<API_KEY>"

setup.ilm.enabled: true

Key settings to validate against your Beats version include the queue type and default path, the monitoring port, and whether filestream is the default input. The max_size value bounds the queue by disk; sizing it too small converts an output outage into input backpressure.

Trust and Data Boundaries

A Beat runs on the source host and inherits that host's file read permissions. If the Beat process can read a log file, anyone who compromises the host can likely read the same data. Treat the Beat host as inside the trust boundary, not an untrusted forwarder.

Output credentials are stored locally, either in the configuration file or the Beats keystore. Use TLS to the output and a least-privilege API key or role rather than a superuser account. A dedicated role that can only write to the intended indices limits the blast radius if the host is compromised.

Because the persistent queue holds unshipped events on local disk, data at rest exists on the source host. If compliance requirements forbid unprocessed data at rest on originating hosts, this design is invalid and a streaming tier that keeps data only in transit becomes necessary.

Operational Checks

Filebeat exposes an HTTP monitoring endpoint, commonly on localhost:5066. Query this from the source host to observe health:

curl -s http://localhost:5066/stats

Look for queue fill percentage, published versus acknowledged events, dropped events, and repeated output backoff. In the logs, watch for messages indicating a stuck or closed file, or that an input has stopped harvesting.

Filebeat tracks progress in a registry file. For filestream inputs, confirm that offsets advance as new lines are read. If offsets stop moving while the log file grows, the harvester—the Filebeat component that reads the file—may be stuck or lacking permissions.

Failure Modes and Design Changes

Several conditions can break this minimal design or necessitate an upgrade to a Logstash/Kafka architecture:

• Registry Corruption: A deleted or corrupted registry can cause re-ingestion from the start or skipped lines. Avoid deleting the registry to "fix" problems; prefer controlled re-reads.
• Log Rotation: copytruncate rotation can lose lines unless close_renamed or renaming is handled correctly by the input configuration.
• Queue Saturation: A full persistent queue blocks inputs and applies backpressure. If the output outage duration exceeds the local disk capacity, data will be lost or inputs will stall.
• Mapping Rejections: Elasticsearch mapping or pipeline rejections appear as failed events. If these require complex conditional logic to fix, move the logic to Logstash.

Move to a Logstash or Kafka tier when you require heavy parsing, enrichment from external databases, fan-out to multiple sinks, or buffering beyond the capacity of a single host's disk.