Scaling Log Collection with Filebeat Autodiscover and Kubernetes Hints

The Problem with Static Log Paths in Kubernetes

In a traditional VM environment, logs live in predictable paths. In Kubernetes, pods are ephemeral, namespaces shift, and containers are spun up and down across a cluster of nodes. If you configure Filebeat with static paths, you face a constant cycle of updating configuration files every time a new service is deployed or a pod is rescheduled.

The solution is Autodiscover. Instead of telling Filebeat where to look, Autodiscover allows Filebeat to watch the Kubernetes API. When a pod starts, Filebeat automatically creates a new input for that container. By using Hints, you shift the control of log collection from the infrastructure team to the application developer, allowing them to define how their logs should be handled via pod annotations.

How Hints-Based Autodiscover Works

Autodiscover operates as a provider that monitors the Kubernetes API for pod events. When a pod matches a specific condition, Filebeat applies a configuration template to that pod's log stream. Hints are a specific type of template that looks for annotations on the pod definition.

This approach decouples the logging agent from the application logic. For example, a developer can signal that a specific pod requires multiline processing for Java stack traces without needing to modify the global Filebeat DaemonSet configuration. This prevents the "global config bloat" that occurs when a single filebeat.yml tries to account for every unique log format in a large cluster.

Implementation Example: Dynamic Multiline Logs

To implement this, Filebeat must be deployed as a DaemonSet with the add_kubernetes_metadata processor enabled. This ensures that every log line is enriched with the pod name, namespace, and labels before it reaches Elasticsearch.

1. Filebeat Configuration

Run this configuration on your Filebeat pods (typically as part of a ConfigMap). This setup tells Filebeat to listen for hints provided in the pod annotations.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*.log

2. Pod Annotation

When deploying your application, add the following annotations to the pod spec. This example tells Filebeat to treat any line starting with something other than a date as a continuation of the previous line (essential for stack traces).

metadata:
  annotations:
    co.elastic.logs/enabled: "true"
    co.elastic.logs/multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    co.elastic.logs/multiline.negate: "true"
    co.elastic.logs/multiline.flush_pattern: '^\d{4}-\d{2}-\d{2}'

Verification Steps

• Deploy: Apply the Filebeat DaemonSet and the annotated pod.
• Check API Access: Ensure the Filebeat ServiceAccount has get, list, and watch permissions for pods.
• Validate: Trigger a stack trace in the application and query Elasticsearch. The entire trace should appear as a single event rather than multiple fragmented lines.

Critical Trade-offs and Limitations

While Autodiscover simplifies scaling, it introduces specific operational risks:

• API Pressure: Filebeat constantly polls the Kubernetes API. In clusters with thousands of pods churning rapidly, this can increase the load on the API server.
• Registry Persistence: Filebeat tracks its position in log files using a registry file. If you do not mount a persistent volume for the registry (e.g., using a hostPath mount), Filebeat will re-read all logs from the beginning of the file every time a Filebeat pod restarts, leading to massive data duplication in Elasticsearch.
• Resource Contention: A single Filebeat instance on a node tails every container on that node. If one container produces logs at an extreme rate, it can saturate the Filebeat queue, potentially delaying logs from other critical services on the same node.

Operational Checklist

Before moving to production, verify your deployment against these requirements:

Requirement	Check	Risk of Failure
Read-Only Mounts	/var/log/containers is read-only	Security vulnerability if writable
Registry Volume	Registry path is persisted to host	Log duplication on restart
RBAC Permissions	ClusterRole allows pod annotation reads	Hints will be ignored

To roll back these changes, remove the hints.enabled: true flag from the Filebeat configuration and revert the pod annotations. This will return Filebeat to its default behavior of treating all logs as single-line events based on the default_config.