Streamlining Container Logs with Filebeat Autodiscover: A Practical Guide

Why Autodiscover Matters for Containerized Workloads

When a fleet of containers scales up or down, keeping a static filebeat.yml that lists every log file becomes a maintenance nightmare. Filebeat Autodiscover solves this by letting Beats automatically detect new containers, apply a matching template, and start shipping logs without any manual re‑configuration.

Core Concepts at a Glance

• Provider: The orchestrator API (Docker, Kubernetes, ECS, etc.) that exposes container metadata.
• Template: A declarative rule that matches metadata and generates an input definition on the fly.
• Input: The actual log source (file, syslog, etc.) that Filebeat reads.
• Runtime Overhead: A small CPU and memory cost for polling the provider and evaluating templates.

Configuring Autodiscover for Docker

Below is a minimal filebeat.yml that enables Autodiscover for Docker and routes logs from any container labeled app=web to an Elasticsearch index named docker-logs.

filebeat:
  prospectors: []

autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: docker
        containers.ids: all

templates:
  - condition:
      contains:
        docker.container.labels.app: web
    config:
      - type: container
        paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
        fields:
          app: web
        fields_under_root: true
        multiline.pattern: "^\d{4}-\d{2}-\d{2}"
        multiline.negate: true
        multiline.match: after

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "docker-logs-%{+yyyy.MM.dd}"

Key points to note:

• The hints.enabled: true line tells Filebeat to automatically discover all running Docker containers.
• The condition.contains matches any container that has the label app=web. Replace this with your own label or regex.
• The paths field uses the container ID to locate the Docker log file on the host.
• All matched logs are forwarded to Elasticsearch with a daily index pattern.

Running the Test Container

Start a container that satisfies the template condition:

docker run -d \
  --label app=web \
  --name my_web_app \
  nginx:latest

Filebeat should now automatically create an input for my_web_app and begin shipping logs. Verify the process:

1. Check Filebeat logs – run filebeat -e -d "autodiscover" -c filebeat.yml and look for a line similar to Created input for container my_web_app.
2. Inspect Elasticsearch – query the docker-logs-* index in Kibana to confirm that logs appear.
3. Validate configuration – use filebeat test config -c filebeat.yml to ensure no syntax errors.

Trade‑offs and Caveats

• Permissions: Filebeat must have access to the Docker socket or Kubernetes API. Without the /var/run/docker.sock mount or proper RBAC, templates won’t match.
• Over‑matching: A broad contains condition can spawn many inputs, increasing memory usage and making log noise harder to sift through.
• Runtime Overhead: Autodiscover polls the provider at a configurable interval (default 30 s). In high‑velocity environments, this overhead can become noticeable.
• Template Complexity: Complex conditions (regex, multiple labels) can be hard to debug. Use filebeat test config and enable debug logging to troubleshoot.

Actionable Checklist for Production

1. Mount the Docker socket or configure Kubernetes RBAC for the Filebeat service account.
2. Define narrow template conditions (e.g., specific label values) to avoid generating unnecessary inputs.
3. Set the autodiscover.providers[].period to a reasonable value (e.g., 60 s) to balance responsiveness with overhead.
4. Enable filebeat setup to pre‑create index templates and mappings in Elasticsearch.
5. Continuously monitor Filebeat logs for autodiscover warnings and adjust templates accordingly.

By following these steps, engineers can deploy Filebeat in containerized environments that scale automatically, while keeping log ingestion reliable and maintainable.