Simplifying Log Shipping with Filebeat Autodiscover: A Practical Guide

Why Autodiscover Matters

When you run dozens of containers or micro‑services, writing a separate Filebeat input for each log source is a maintenance nightmare. Every new container, image update, or environment change forces you to edit configuration files, restart Filebeat, and risk mis‑configuring a stream. Filebeat’s autodiscover feature solves this by letting a single configuration file generate inputs on the fly based on container or host metadata.

How Autodiscover Works

Autodiscover listens to a metadata source—Docker, Kubernetes, or a custom API—and applies templates that match labels, annotations, or other attributes. When a match is found, Filebeat automatically creates an input that starts reading the container’s log file and forwards it to the configured output.

Key points:

• Templates are written in Filebeat’s standard config language.
• Matching is done by simple key/value look‑ups (e.g., docker.labels.foo == "bar").
• Generated inputs are added to Filebeat’s runtime configuration; no manual file edits are required.

Step‑by‑Step Example: Shipping Docker Logs with a Label

1. Prerequisites
   • Filebeat 8.x installed on the host.
   • Docker socket access (or Docker API endpoint) granted to the Filebeat user.
   • Container(s) running with a custom label, e.g., com.example.log=true.
2. Enable Autodiscover in filebeat.yml

   filebeat.autodiscover:
     providers:
       - type: docker
         hints.enabled: true
         hints.default_config.enabled: false
   

   This tells Filebeat to watch Docker for containers with the hints.enabled flag. The hints.default_config.enabled: false line prevents Filebeat from creating a generic input for every container.

3. Create a Template for Labeled Containers

   filebeat.autodiscover:
     providers:
       - type: docker
         hints.enabled: true
         hints.default_config.enabled: false
         templates:
           - condition:
               equals:
                 docker.labels.com.example.log: "true"
             config:
               - type: container
                 paths: "/var/lib/docker/containers/*/*.log"
                 multiline.pattern: ^\d{4}-\d{2}-\d{2}
                 multiline.negate: true
                 multiline.match: after
   

   The condition block matches containers that carry the specified label. When a match occurs, Filebeat creates a container input that reads the container’s JSON log file and applies a multiline pattern to group stack traces.

4. Verify the Configuration

   sudo filebeat test config -e
   

   Look for lines such as autodiscover: template matched for container ... in the output. This confirms the template is syntactically correct and will be applied when a matching container starts.

5. Run Filebeat

   sudo systemctl start filebeat
   sudo systemctl status filebeat
   

   Check the logs for entries like autodiscover: created input for container .... If you see the input, logs from the container should now be forwarded to your Elasticsearch or Logstash endpoint.

6. Test with a Sample Container

   docker run -d --label com.example.log=true nginx:latest
   

   After a few seconds, use curl http://localhost:9200/_cat/indices (or your monitoring UI) to confirm the index is receiving data.

Trade‑offs and Limitations

• Permissions: Filebeat must have read access to the Docker socket or Kubernetes API. On Kubernetes, this means setting up a ServiceAccount with the cluster-reader role or similar. Granting too many permissions can expose sensitive metadata.
• Duplicate Streams: If multiple templates match the same container, Filebeat will create multiple inputs, potentially causing duplicate log entries. Carefully design template conditions to be mutually exclusive.
• Runtime Overhead: Autodiscover polls the metadata source at a configurable interval (default 30 s). The added CPU and network usage is usually negligible but can add up in very large clusters.
• Version Support: Autodiscover is available from Filebeat 7.0 onward. On older releases, you’ll need to manually edit the filebeat.yml for each container.

Actionable Next Steps

• Audit your container labels or Kubernetes annotations to ensure they are unique and consistently applied.
• Write a small test harness that spins up a container with the required label and verifies that Filebeat creates the input and forwards logs.
• Implement a filebeat.test.config CI step that runs against your production filebeat.yml to catch template syntax errors before deployment.
• Monitor Filebeat’s autodiscover logs for any unexpected template matches or failures.
• Consider adding custom processors (e.g., drop_fields) in your template to filter out noisy logs from specific services.

By leveraging autodiscover, you turn a static, error‑prone configuration into a dynamic, maintainable system that scales with your container fleet. Once the templates are in place, adding a new micro‑service is as simple as attaching the right label—no Filebeat restart required.