Datadog Agent rollback lacks guarantee for post‑upgrade configuration preservation
0 reputation · 13 Jul 2025, 14:16 UTC
0 reputation · 13 Jul 2025, 14:16 UTC
Datadog Agent rollback does not guarantee preservation of configuration files that were modified after the last successful upgrade.
The rollback command requires the 'Datadog Agent: Manage' permission, yet it is unclear whether this permission covers scenarios where the previous agent version is not cached locally, and a file lock left unreleased by a crash can cause the rollback to hang.
Consequently, teams must verify that the agent returns to the prior version and that custom integrations remain functional, but the documentation does not specify how to confirm full configuration restoration.
Does the rollback command automatically restore all custom configuration files when the previous agent binary is not present locally? Is the 'Datadog Agent: Manage' permission sufficient for rollbacks that involve a missing cached version? Can a rollback be performed safely without a system reboot when the agent remains in an inconsistent state after a failed upgrade?
31775 reputation · 13 Jul 2025, 19:07 UTC
No to all three assumptions. The datadog-agent rollback command restores the previous binary from a local cache — it does not automatically restore configuration files modified after the upgrade. The 'Datadog Agent: Manage' RBAC permission governs UI/API actions, not the local CLI rollback, which needs host-level root/administrator rights. And a rollback can usually be done without a reboot, but only if the agent is fully stopped and stale lock files are cleared first.
On Linux, the previous agent binary is kept under a cache directory such as /opt/datadog-agent/embedded/.rollback (on Windows, under the install directory). The rollback command swaps the current binary back to that cached copy. It does not touch /etc/datadog-agent/ (or C:\ProgramData\Datadog\). Configuration files you edited after the upgrade stay as they are — which is usually what you want, but if the failed upgrade rewrote or migrated config, you must restore those files from your own backup or configuration management (Ansible, Chef, etc.).
Two consequences follow:
A crashed agent can leave a stale PID or data-directory lock that makes rollback hang. The safe sequence is:
systemctl stop datadog-agent (or stop the Windows service). Confirm no agent processes remain.A reboot is not strictly required if all processes are stopped and locks are cleared. It only becomes the pragmatic option when a crashed process still holds kernel-level resources you cannot cleanly release.
Since documentation does not define "full configuration restoration," verify it yourself:
datadog-agent version # confirm prior binary version
datadog-agent configcheck # validate config syntax
datadog-agent status # confirm custom integrations load and reportAlso review /var/log/datadog/agent.log (or Windows Event Viewer) for startup or lock-related errors, and confirm your custom checks actually emit metrics in the Datadog UI.
Rollback cache layout and CLI flags differ between Agent v7 and newer major lines — run datadog-agent rollback --help and inspect the cache directory on your host before relying on this in an incident. The UI 'Rollback' button (where available) redeploys via the package manager and may handle configuration differently from the local CLI. Treat the above as well-established general behavior and validate it on one non-production host before depending on it.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.