Choosing a Datadog Agent Deployment Model in Kubernetes: DaemonSet vs Sidecar vs Host Process

Decision: Which Deployment Model for the Datadog Agent?

When you want to monitor a Kubernetes cluster with Datadog, you first decide how to run the Agent. Three supported models exist:

• DaemonSet – one Agent pod per node.
• Sidecar – an Agent container inside the same pod as an application.
• Host process – the Agent runs as a privileged container on the host, exposing kernel‑level metrics.

Each model has distinct constraints, coverage, and operational overhead. This guide helps you pick the right one for your cluster size, security posture, and observability needs.

Constraints to Consider

• Cluster size – How many nodes will host Agent pods?
• Observability scope – Do you need node‑level metrics, container logs, or kernel stats?
• Resource limits – CPU, memory, and storage budgets per node.
• Security policies – Are privileged containers allowed?
• Operational simplicity – How much configuration per application versus cluster‑wide?

Supported Options in a Compact Table

Model	Coverage	Resource Footprint	Security Impact	Configuration Complexity
DaemonSet	All nodes, all containers, full metrics, optional logs.	One pod per node; moderate CPU/memory.	Runs as non‑privileged by default; minimal risk.	Cluster‑wide ConfigMap; no per‑pod changes.
Sidecar	Only the pod it shares with; limited to its namespace.	Per‑application pod; small overhead.	Non‑privileged; no host access.	Configuration per application; requires per‑pod annotations.
Host process	Full node + kernel metrics, all containers.	Large pod per node; uses host resources.	Requires privileged: true and hostPath mounts.	Cluster‑wide ConfigMap; but needs privileged setup.

Trade‑Offs Explained

Coverage vs. Overhead
DaemonSet offers the broadest coverage without needing application changes, but the per‑node pod count can strain clusters with thousands of nodes. Sidecar keeps the Agent lightweight and ties it to a single application, which is ideal for micro‑service environments where you only care about that service’s metrics. Host process gives the richest data set (e.g., kubelet metrics, kernel counters) but at the cost of privileged containers and higher resource consumption.

Security
Running as a non‑privileged DaemonSet or sidecar reduces attack surface. Host process exposes the host filesystem and kernel, which is risky if the Agent is compromised. Always store the Datadog API key in a Kubernetes Secret and reference it via environment variables.

Operational Simplicity
DaemonSet requires a single ConfigMap and a DaemonSet manifest. Sidecar demands per‑pod annotations or init containers to mount the same config, complicating CI/CD pipelines. Host process adds privileged flags and hostPath mounts, increasing the risk of misconfiguration.

Concrete Implementation: DaemonSet Example

Below is a minimal, production‑ready DaemonSet that collects node metrics and container logs. Replace YOUR_DATADOG_API_KEY with a reference to a Kubernetes Secret.

apiVersion: v1
kind: Secret
metadata:
  name: datadog-api-key
  namespace: monitoring
stringData:
  DD_API_KEY: <YOUR_DATADOG_API_KEY>
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: datadog-agent-config
  namespace: monitoring
data:
  datadog.yaml: |
    api_key: ${DD_API_KEY}
    logs_enabled: true
    listeners:
      - name: dogstatsd
        port: 8125
    apm_config:
      enabled: true
  conf.d/mysql.d/conf.yaml: |
    init_config: {}
    instances:
      - server: localhost
        port: 3306
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: datadog-agent
  namespace: monitoring
  labels:
    app: datadog-agent
spec:
  selector:
    matchLabels:
      app: datadog-agent
  template:
    metadata:
      labels:
        app: datadog-agent
    spec:
      serviceAccountName: datadog-agent
      containers:
        - name: agent
          image: datadog/agent:7.51.0
          env:
            - name: DD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-api-key
                  key: DD_API_KEY
          volumeMounts:
            - name: config
              mountPath: /etc/datadog-agent
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 200m
              memory: 256Mi
      volumes:
        - name: config
          configMap:
            name: datadog-agent-config
        - name: varlog
          hostPath:
            path: /var/log
            type: Directory
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
            type: Directory

Key points:

• All log files are mounted from /var/log and /var/lib/docker/containers, enabling the Agent to read container logs without sidecars.
• Custom integrations (e.g., MySQL) live in conf.d via the ConfigMap.
• Resource limits keep the Agent from starving workloads.

Verification Steps

1. Deploy the DaemonSet
   Run kubectl apply -f datadog-agent.yaml in the monitoring namespace.
2. Confirm pod status
   Execute kubectl get pods -n monitoring -l app=datadog-agent. All pods should be Running.
3. Check Agent logs
   Run kubectl logs -n monitoring -l app=datadog-agent -c agent | grep -i 'log ingest' to see that logs are being forwarded.
4. Validate metrics in Datadog
   Log in to the Datadog UI, navigate to Infrastructure > Hosts, and verify that each node appears with metrics.
5. Verify log stream
   In Datadog, open the Logs Explorer and filter by source:container to confirm container logs are visible.
6. Monitor resource usage
   Run kubectl top pod -n monitoring -l app=datadog-agent to ensure CPU/memory stay within the limits.
7. Rolling update test
   Patch the image tag to a newer version and observe the DaemonSet performing a rolling update: kubectl set image ds/datadog-agent agent=datadog/agent:7.52.0. Verify no pod crashes and metrics/logs continue.

All verification steps should succeed without manual intervention, confirming that the DaemonSet meets the cluster’s observability requirements.

Conclusion

For most Kubernetes environments, a DaemonSet strikes the best balance: full node coverage, minimal per‑application changes, and a manageable resource footprint. Sidecars are useful when you only need to monitor a subset of services or wish to keep the Agent isolated from the node. Host process mode is reserved for scenarios that demand kernel‑level metrics and is only recommended when the security model allows privileged containers.

Practical Checklist

• Is your cluster large? Consider node selectors or resource quotas to limit Agent pods.
• Do you need kernel metrics? Host process may be required.
• Are you constrained by security policies? Avoid privileged containers.
• Do you want minimal configuration drift? Use DaemonSet with ConfigMap.

Apply the decision guide, set up the chosen model, and follow the verification steps to ensure reliable, cost‑effective monitoring with Datadog.