Choosing Between Datadog Agent v6 and v7 in Kubernetes

Decision and constraints

You need to run a Datadog Agent in a Kubernetes cluster (v1.27+) that collects APM traces, process metrics, and logs while keeping CPU and memory overhead low. The decision is which Agent major version to use: v6 (stable) or v7 (newer with eBPF‑based network metrics).

Supported options

Feature	Agent v6	Agent v7
Typical CPU usage	~5 % of a core	~3 % of a core
Typical Memory usage	~150 MiB	~120 MiB
APM support	Full	Full
Process monitoring	Enabled	Enabled
Log collection	Enabled	Enabled
eBPF network metrics	Not available	Available (requires kernel ≥ 4.15)
Release status	Stable, long‑term support	Newer, may contain occasional beta quirks

Trade‑offs

• Agent v6 offers proven stability and a lower risk of upgrade‑related issues, but it cannot provide eBPF‑derived network insights such as per‑socket latency or drop statistics.
• Agent v7 reduces CPU and memory footprint and adds eBPF network metrics, yet it requires a Linux kernel with eBPF support and may need updated kernel headers or extra privileges on some node images.

Implementation steps

1. Create a Kubernetes secret that holds your Datadog API key (if you do not already have one):

   kubectl create secret generic datadog-secret --from-literal=api-key= --namespace 
   

2. Save the following DaemonSet manifest as datadog-agent-daemonset.yaml, choosing the image tag that matches the version you want to deploy:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: datadog-agent
  labels:
    app: datadog-agent
spec:
  selector:
    matchLabels:
      app: datadog-agent
  template:
    metadata:
      labels:
        app: datadog-agent
    spec:
      serviceAccountName: datadog-agent
      containers:
        - name: datadog-agent
          image: datadog/agent:   # e.g., v6.20.0 or v7.45.0
          env:
            - name: DD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-secret
                  key: api-key
          volumeMounts:
            - name: dockersocket
              mountPath: /var/run/docker.sock
            - name: procfs
              mountPath: /host/proc
              readOnly: true
            - name: cgroupsfs
              mountPath: /host/sys/fs/cgroup
              readOnly: true
          # For v7 enable eBPF network metrics
          # Uncomment the following line if you chose v7 and your nodes support eBPF
          # - name: DD_EBPF_ENABLED
          #   value: 'true'
      volumes:
        - name: dockersocket
          hostPath:
            path: /var/run/docker.sock
        - name: procfs
          hostPath:
            path: /proc
        - name: cgroupsfs
          hostPath:
            path: /sys/fs/cgroup

1. Apply the DaemonSet:

   kubectl apply -f datadog-agent-daemonset.yaml -n 
   

2. If you are using Agent v7, edit the DaemonSet to set the environment variable DD_EBPF_ENABLED=true (or add it to the agent’s datadog.yaml via a ConfigMap) and re‑apply.
3. Monitor the rollout for CrashLoopBackOff or pending pods:

   kubectl rollout status daemonset/datadog-agent -n 
   

Validation

1. Identify a running agent pod:

   kubectl get pods -n  -l app=datadog-agent
   

2. Execute the info command inside the pod and check the version string:

   kubectl exec -it  -n  -- datadog-agent info
   

   Look for a line similar to: Agent version: 1:7.45.0 (or the v6 equivalent).
3. In the same output verify that the collections you enabled are reported as active, for example: APM: enabled: true Process Agent: enabled: true Logs: enabled: true If you enabled eBPF, you should also see a section like: eBPF: enabled: true
4. Check resource usage to ensure it stays within the expected range:

   kubectl top pods -n  -l app=datadog-agent
   

   The CPU column should be close to the predicted percentage (≈5 % for v6, ≈3 % for v7) and memory under the listed MiB values.
5. Finally, confirm in the Datadog UI:
   • Infrastructure → Containers list shows the host with the correct agent version.
   • APM → Traces appears within a few minutes after sending test traffic.
   • Live Processes and Logs panels populate data.

Limitations and practical checks

Agent v7 will fall back to non‑eBPF mode or fail to start on nodes with a kernel older than 4.15 or without eBPF support. You can detect this by looking for the line eBPF: disabled (kernel version too old) in the info output. If you see such a message, either upgrade the node image or stay with Agent v6 for those nodes.

Changing from v6 to v7 also changes the default location of some configuration files; any ConfigMap that mounts /etc/datadog-agent/conf.d must be updated to match the new path, otherwise the agent may silently ignore custom checks.

Before a cluster‑wide rollout, test the DaemonSet in a staging namespace with a mix of node types. Verify that no pod enters CrashLoopBackOff and that the UI shows the expected metrics.