Deploying Filebeat Modules: A Practical Architecture Note

Problem Statement

Organizations often need to ingest a variety of log sources into Elasticsearch or Logstash. Writing custom Filebeat inputs, processors, and pipelines for each source is time‑consuming and error‑prone. Filebeat modules bundle pre‑configured ingest pipelines and parsers for common log types, enabling rapid deployment while keeping the configuration manageable. This architecture note outlines the functional requirements, minimal design, trust boundaries, operational checks, failure modes, and conditions that would prompt a redesign when using Filebeat modules.

Functional Requirements

• Rapid onboarding – Enable a log source with a single module activation.
• Least‑privilege logging – Only the required module should run, limiting the attack surface.
• Secure transport – All data to Elasticsearch or Logstash must use TLS.
• Extensibility – Ability to inject custom processors without breaking module logic.
• Version compatibility – Module files must match the Filebeat release.

Minimal Design

The smallest suitable design consists of a single Filebeat instance per host, with modules enabled only for the required log sources. The core components are:

• Filebeat binary – runs as a system service.
• Module configuration – placed in /etc/filebeat/modules.d/.
• Output section – TLS‑enabled Elasticsearch or Logstash endpoint.
• Optional autodiscover – for dynamic environments like Docker or Kubernetes.

Below is a minimal filebeat.yml that enables the nginx module and sends data to a secured Elasticsearch cluster.

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: false
    # Filebeat modules provide their own inputs.

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

output.elasticsearch:
  hosts: ["https://es01.example.com:9200"]
  username: "elastic"
  password: "changeme"
  ssl.certificate_authorities: [ "/etc/filebeat/certs/ca.crt" ]
  ssl.verification_mode: full

# Optional: enable autodiscover for Docker
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      templates:
        - condition.equals: {"docker.container.image": "nginx:latest"}
          config:
            - module: nginx
              var.paths: ["/var/lib/docker/containers/**/nginx.log"]

Each module resides in /etc/filebeat/modules.d/.yml and contains its own input, processors, and ingest pipeline definition. Enabling a module automatically registers the required input and processor chain.

Trust and Data Boundaries

• File permissions – Module files should be owned by root and readable only by the Filebeat user to prevent tampering.
• Network isolation – The Filebeat service should run on a host with outbound access only to the Elasticsearch/Logstash cluster, preferably via a dedicated VPN or VPC endpoint.
• Least‑privilege credentials – Use dedicated service accounts for Filebeat with only the write role on the target index.
• Logging of sensitive data – Modules that parse authentication logs (e.g., auditd) should be enabled only on hosts that are hardened and monitored for data exfiltration.

Operational Checks

1. Configuration validation – Run filebeat --configtest -e -c /etc/filebeat/filebeat.yml as the Filebeat user. Expected output: "Config OK" without errors.
2. Module load verification – After starting Filebeat, check the logs (/var/log/filebeat/filebeat.log) for lines like "Module nginx enabled". Verify that the module’s ingest pipeline exists in Elasticsearch: GET _ingest/pipeline/nginx.
3. Test ingestion – Create a synthetic log file in the module’s path (e.g., /var/log/nginx/access.log) and watch the document appear in Elasticsearch. Use curl -XGET "https://es01.example.com:9200/_search?q=nginx" -u elastic:changeme.
4. TLS verification – In the Filebeat logs, look for messages indicating "SSL handshake succeeded". In Elasticsearch, confirm the certificate fingerprint matches the one in /etc/filebeat/certs/ca.crt.

Failure Modes and Mitigation

Failure Mode	Cause	Mitigation
Module config error	Syntax error in modules.d/*.yml	Run filebeat --configtest before restart; enable logging.level: debug to capture details.
Missing input plugin	Module relies on an input that has been disabled globally	Ensure filebeat.inputs is not overriding module inputs; set enabled: false for generic inputs only.
Output TLS failure	Certificate mismatch or revoked CA	Update ssl.certificate_authorities; rotate certificates; verify via openssl s_client -connect es01.example.com:9200.
Pipeline compatibility issue	Upgrading Filebeat without updating module pipelines	After upgrade, run filebeat modules enable --list and manually update any modules flagged as incompatible.
Data leakage	Module exposes sensitive logs without proper access controls	Use index templates with strict read/write permissions; separate indices for sensitive sources.

Design Re‑evaluation Triggers

• High ingestion volume – If a module generates >1 M events/s, consider routing through Logstash for buffering and advanced processing.
• Multi‑tenant environment – When multiple teams share the same Filebeat instance, custom processors may conflict; isolate per‑team modules or use separate Filebeat services.
• Custom enrichment required – If the module’s default processors do not meet business rules, inject additional processors in the module config or create a wrapper module.
• Compliance changes – New regulations may require field masking or redaction; update the ingest pipeline or add a drop_fields processor.
• Upgrade path constraints – When a new Beats release deprecates an input type used by a module, the design must shift to a newer module or custom input.

Practical Checklist for Production Deployment

• Verify module version compatibility with the Filebeat release.
• Restrict file permissions: chown root:filebeat /etc/filebeat/modules.d/*.yml, chmod 640.
• Configure TLS with a dedicated CA and rotate certificates annually.
• Enable logging.level: warning in production to reduce noise while still capturing errors.
• Set up alerting on Filebeat logs for failed connections or pipeline errors.
• Document the module enablement matrix for audit purposes.

Conclusion

Filebeat modules provide a powerful, low‑maintenance way to ingest structured logs into Elasticsearch or Logstash. By following the minimal design, enforcing trust boundaries, conducting thorough operational checks, and anticipating failure modes, teams can deploy a reliable, secure log ingestion pipeline that scales with their needs. When volume spikes, compliance demands change, or custom enrichment is required, revisit the design to incorporate Logstash buffering, additional processors, or isolated Filebeat instances.