Using Elastic Beats Autodiscover to Dynamically Harvest Logs in Kubernetes

Problem: Keeping log collection in sync with changing workloads

In a Kubernetes cluster, pods are created and destroyed constantly. If you manually configure a Filebeat DaemonSet to watch a static set of log paths, you either miss logs from newly‑scaled services or waste resources harvesting from terminated pods. The goal is to let the observability agent discover each pod’s logging configuration automatically and start/stop harvesters as the pod lifecycle changes.

How Beats autodiscover works

The autodiscover subsystem runs inside the Beats agent and watches the Kubernetes API for pod events. When a pod matches a condition you define (usually a label or annotation), autodiscover renders a template for a Beat module (e.g., the Filebeat nginx module) using variables that are replaced with pod‑specific data such as the container’s IP, port, or log file path. When the pod terminates, the corresponding Beat instance is stopped cleanly, preventing orphaned harvesters and duplicate events.

Key concepts:

• Condition – a match rule (equals, regexp, range) that can be combined with AND/OR logic.
• Template variables – placeholders like ${data.host}, ${data.port}, or ${data.labels.app} that are substituted at runtime.
• Graceful stop – on pod deletion, autodiscover sends a stop signal to the launched Beat, ensuring no stale harvesters remain.

Worked example: Harvesting nginx logs with Filebeat

Below is a minimal Filebeat DaemonSet manifest that enables autodiscover for pods labeled app=nginx. The configuration assumes you have already installed the Elastic Stack and have appropriate RBAC for the service account running Filebeat.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.12.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          subPath: filebeat.yml
        - name: varlog
          mountPath: /var/log
        - name: docker-socket
          mountPath: /var/run/docker.sock
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    k8s-app: filebeat
Data:
  filebeat.yml: |
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: false
          templates:
            - condition:
                equals:
                  kubernetes.labels.app: "nginx"
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/var/log/nginx/access.log*"]
                  error:
                    enabled: true
                    var.paths: ["/var/log/nginx/error.log*"]
    output.elasticsearch:
      hosts: ["${ELASTICSEARCH_HOST}:9200"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
    setup.kibana:
      host: "${KIBANA_HOST}:5601"

What happens when you apply this manifest?

1. The Filebeat pod starts on each node and begins watching the Kubernetes API.
2. When a deployment creates a pod with the label app=nginx, autodiscover matches the equals condition.
3. The template is rendered: ${NODE_NAME} becomes the node where the pod runs, and the paths /var/log/nginx/access.log* and /var/log/nginx/error.log* are interpreted relative to the pod’s filesystem (mounted via the host’s /var/log volume).
4. Filebeat launches an nginx module harvester for that container and starts sending lines to Elasticsearch.
5. If the pod is deleted, autodiscover receives the deletion event and stops the harvester, ensuring no stale data is ingested.

Trade‑offs and limitations

• RBAC requirements – The service account must be able to list and watch pods, namespaces, and nodes. Missing permissions cause silent skips; you’ll see no autodiscover logs even though pods are present.
• Template debugging – Complex conditions or nested variables can be hard to troubleshoot. Enabling debug output (filebeat -d "*") increases verbosity and may affect performance in high‑scale clusters.
• Resource overhead – Each matched pod spawns a Beat instance (or module) inside the Filebeat container. For very dense workloads, consider using the hints‑based approach or consolidating multiple logs into a single module to reduce the number of active harvesters.
• Version compatibility – The autodiscover API is stable from Beats 7.0 onward, but certain module‑specific variables (e.g., var.paths) may change between minor releases. Consult the module documentation for the exact variable names supported in your Beat version.

Actionable next steps

1. Create a dedicated service account for Filebeat with the minimal RBAC rules (pods: list,watch, namespaces: get,list,watch, nodes: get,list,watch).
2. Deploy the DaemonSet above, adjusting the Elasticsearch host, credentials, and any module‑specific variables to match your environment.
3. Verify the setup by checking Filebeat logs for lines like autodiscover: started harvester for container xxx when an nginx pod appears and a corresponding stopped message when it disappears.
4. Confirm that logs are indexed under the expected pattern (e.g., filebeat-nginx-*) and that no entries are generated for pods lacking the app=nginx label.