Filebeat Autodiscover with Docker: Minimal Architecture for Dynamic Log Shipping

Requirements

To ship container logs dynamically you need:

• A host with Docker Engine running.
• Filebeat installed (non‑root user) with the docker input enabled.
• A reachable Logstash instance that accepts Beats input over TLS (mutual auth).
• Optional: Elasticsearch for storage and indexing.

Smallest Suitable Design

The minimal deployment consists of a single Filebeat process on the Docker host, the Docker input plugin, and a Logstash pipeline. No sidecar containers or extra agents are required.

Filebeat configuration (example)

filebeat.inputs:
- type: docker
  containers.ids:
  - '${data.docker.container.id}'
  processors:
  - add_docker_metadata:
      host: "unix:///var/run/docker.sock"
      match: ids
      match_priority: "latest"
output.logstash:
  hosts: [":5044"]
  ssl.certificate_authorities: [""]
  ssl.certificate: ""
  ssl.key: ""

Place the file at /etc/filebeat/filebeat.yml and start Filebeat as a non‑root user that belongs to the docker group (or has ACL read access to /var/lib/docker/containers).

Logstash pipeline (example)

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => ""
    ssl_key => ""
  }
}
filter {
  # parse Docker JSON logs if needed
  json {
    source => "message"
    target => "docker"
  }
}
output {
  elasticsearch {
    hosts => [":9200"]
    index => "filebeat-%{+YYYY.MM.dd}"
    user => ""
    password => ""
  }
}

Trust and Data Boundaries

• Docker API trust: Filebeat talks to the Docker daemon via the Unix socket (/var/run/docker.sock). The socket grants the same privileges as the Docker group; limit Filebeat’s group membership to only what is needed.
• Log transport trust: Mutual TLS ensures Filebeat authenticates Logstash and vice‑versa. Certificates must be rotated in sync; a mismatch breaks the connection.
• Data boundary: Filebeat reads only the log files under /var/lib/docker/containers. It does not access container images or runtime metadata beyond what the Docker API provides via labels.

Operational Checks

1. Verify Filebeat health: curl -s http://localhost:5066/health | jq . – look for status: "green" and non‑zero events.published.
2. Check Docker plugin discovery: curl -s http://localhost:5066/state | jq .module.docker – should list discovered container IDs matching running containers.
3. Confirm Logstash pipeline status: curl -s http://localhost:9600/_node/pipelines?pretty – ensure state: "started" and events.in matches Filebeat output.
4. Inspect Elasticsearch index: GET filebeat-*/_count – verify document count grows with log traffic.

Failure Modes

• Log rotation before read: If a container rotates its log file faster than Filebeat can harvest, lines may be lost. Mitigate by increasing close_inactive or using ignore_older settings.
• Network partitions: TCP back‑pressure can fill Filebeat’s internal queue; monitor queue.max_events via the health endpoint.
• Permission changes: Altering ACLs on /var/lib/docker/containers removes Filebeat’s read access, causing silent drops. Use immutable ACLs or run Filebeat as root only if unavoidable.
• Docker API rate limits: Frequent container start/stop can exceed the daemon’s request quota, leading to missed discovery events. Watch Docker daemon metrics (docker info APIRequests) and enable docker.api.period throttling in Filebeat if needed.

When the Design Should Change

• If sustained log volume exceeds ~100 GB per day per host, the Filebeat‑to‑Logstash link may become a bottleneck; consider inserting a Kafka buffer or using Logstash‑only ingestion with persistent queues.
• When compliance requires encryption at rest for logs before they reach Elasticsearch, add an intermediate encrypted storage layer (e.g., encrypted S3 bucket) and adjust the pipeline accordingly.
• If you need to ship logs from multiple Docker hosts with differing security zones, deploy a dedicated Filebeat agent per zone and centralize Logstash behind a load balancer.