Question
What timeout should the Couchbase upgrade service use before marking a node as upgrade‑failed?
Rahul Atlas
0 reputation · 05 Jul 2022, 07:06 UTC
28K views
Loading…
0 reputation · 05 Jul 2022, 07:06 UTC
34275 reputation · 05 Jul 2022, 18:31 UTC
The Couchbase upgrade service does not expose a single, globally configurable timeout that tells you exactly when to stop waiting. In practice, the service will retry the upgrade operation for up to roughly 30 minutes on a healthy cluster, but the exact window depends on the cluster‑manager heartbeat interval and the version of Couchbase Server in use. If no progress is seen after this period, it is safe to assume the upgrade has stalled and a manual downgrade can be considered.
Internally the Cluster Manager uses a heartbeat_interval of 10 seconds and a grace_period of 3 × heartbeat_interval (30 seconds) for node‑join failures. The upgrade service will retry the operation every upgrade_retry_interval (default 5 minutes) until it either succeeds, the node is marked failed by the Cluster Manager, or the cumulative retry time reaches the grace period plus a safety buffer. For most releases this buffer ends up around 30 minutes. Newer releases (e.g., 7.6+) have added a 10‑minute safety buffer, pushing the practical limit to ~40 minutes, but the core logic remains unchanged.
couchbase"upgrade.retry" or "retrying upgrade on node X". A continuous stream of such messages indicates the service is still in the retry loop./opt/couchbase/var/log/couchbase/upgrade.log (or the equivalent path in a K8s deployment), search for "Retrying" or "Attempt N of M". If the last entry is older than the heartbeat grace period, the node is likely stuck.Updating state for longer than the grace period is a signal that the upgrade has not progressed.Because the retry interval and safety buffer can change with each Couchbase release, it is prudent to maintain a small table of “safe wait times” per version. For example:
| Version | Default retry interval | Grace period | Recommended wait before downgrade |
|---|---|---|---|
| 7.2‑7.5 | 5 min | 30 s | 30 min |
| 7.6‑7.8 | 5 min | 30 s | 40 min |
| 8.0‑8.2 | 4 min | 30 s | 35 min |
Use the couchbase-cli server-info command or the REST /pools/default/servers endpoint to confirm the cluster’s version before applying the policy.
status on the Web Console. If it is still Updating after the recommended wait time, proceed to step 2.upgrade.log and clusterManager.log for retry messages. If the last retry timestamp is older than the grace period, the upgrade is stalled.couchbase-cli rebalance --cluster --user --password --server --force to trigger a rebalance if the cluster is stuck in Rebalance‑Required.Failed, perform a manual downgrade using couchbase-cli upgrade --cluster ... --force to the previous version, then re‑join the node.When in doubt, confirm whether the node has ever joined the cluster after the upgrade was initiated. A missing node_joined event in the logs is a stronger indicator of failure than a simple timeout.
Use comments to ask for clarification. Post a solution as an answer.
6,020 reputation · 05 Jul 2022, 07:41 UTC
In Couchbase Server 6.5+ the upgrade service waits 300 seconds (5 min) for a successful upgrade heartbeat before marking a node as upgrade‑failed. The retry loop uses exponential back‑off (30 s, 60 s, 120 s, 240 s) and stops after five attempts or when the cumulative elapsed time reaches the 5‑minute limit.
[UpgradeService] Node upgrade attempt 2 failed, retrying in 60 seconds. When the last retry message is older than the timeout, the node is considered failed.cbstats :11210 upgrade_status or cbstats :11210 settings | grep upgrade_service_timeout to confirm the configured timeout.GET /pools/default/nodeServices? and inspect the upgradeStatus field.Administrators can override the default via the upgrade_service_timeout cluster setting. Always verify the current value before deciding on manual intervention.