Do Couchbase alert thresholds survive node removal without a full rebalance?

Answer

Alert definitions persist after a node is failed over and removed as long as the cluster’s config service (Raft‑based since Couchbase Server 6.5) retains a quorum of nodes. If a quorum is lost, the definitions can disappear; otherwise they survive without a full rebalance.

Confirmed facts

• Alert configurations are stored in the cluster‑wide config service and replicated to all nodes.
• In Couchbase 6.5+ the config service uses Raft, so the data survives node removal when a majority of config‑service replicas remain.
• Prior to 6.0 the alert config lived only in the local cluster configuration file and could be lost if the node holding the master copy was removed without a rebalance.
• A per‑alert throttle setting overrides the global throttle for that alert; the two are not additive.
• Exporting via GET /pools/default/alerts returns a JSON payload that can be re‑applied with a POST to the same endpoint to restore alerts after topology changes.

Likely explanation

When a node is removed, the config service continues to run on the remaining nodes. As long as a majority of those nodes are still reachable, the Raft log containing the alert definitions stays intact and is available to the surviving nodes. If the removal drops the config service below a majority, the service loses leadership and the alert config becomes unavailable until the service is restored or a new quorum is formed.

Steps for this case

1. Before any node removal, back up the alert definitions: curl -u admin:password http://:8091/pools/default/alerts -o alerts-backup.json.
2. Perform the node failover/removal as needed.
3. After the cluster stabilizes, verify the definitions are still present: curl -u admin:password http://:8091/pools/default/alerts and compare to the backup.
4. If the definitions are missing, restore them: curl -u admin:password -X POST http://:8091/pools/default/alerts -H 'Content-Type: application/json' -d @alerts-backup.json.

Missing diagnostic detail

To give a definitive recommendation, please confirm the exact Couchbase Server version you are running (e.g., 6.0.x, 6.5.x, 7.x). The persistence behavior differs between pre‑6.0 and 6.5+ releases.