Aerospike cluster upgrade: pause or continue when a node fails to start

Aerospike Upgrade Failure: Immediate Decision

When a node refuses to start after a binary upgrade, the cluster does **not** automatically pause. It marks the node offline and continues to serve traffic from the remaining nodes. The safety of this continuation depends on two factors:

• Replication factor of the namespaces – if it is 1, a single node loss means temporary data unavailability.
• Protocol version compatibility – a mixed‑version cluster is only safe when all nodes expose the same protocol API.

Should the cluster pause?

Pause only if:

1. The replication factor is 1 (or the cluster cannot tolerate a single node loss).
2. The failed node’s protocol version differs from the healthy nodes.
3. Clients report protocol errors or degraded performance.

Otherwise, you can keep the cluster running while you address the failed node.

Safe Downgrade & Re‑join Sequence

Follow this minimal‑impact workflow. All commands are safe to run on an individual node; avoid stopping the entire cluster unless the replication factor is 1.

1. Stop the node gracefully (if it’s still running partially):

   sudo systemctl stop aerospike

2. Rollback binaries – replace the upgraded files with the previous version’s packages or tarball.

   sudo apt-get install aerospike=previous_version   # Debian/Ubuntu
   # or
   sudo yum install aerospike-previous_version.rpm   # RHEL/CentOS
   

3. Restore the original asd.conf if it was altered during the upgrade. Keep a backup of the pre‑upgrade file.

   sudo cp /opt/aerospike/conf/backup/asd.conf.OLD /opt/aerospike/conf/asd.conf

4. Verify configuration:

   asinfo -v config

5. Restart the node:

   sudo systemctl start aerospike

6. Check node status on a healthy peer:

   asinfo -v node-status

7. Re‑join the cluster – the node should automatically advertise itself. If it does not, force a join:

   asinfo -v "rejoin=1"

8. Verify replication health:

   asinfo -v replication-status

Mixed‑Version Operation

Protocol compatibility is the key. Aerospike’s protocol version is exposed via asinfo -v protocol-version. If the upgraded node reports a different value than the rest, clients may see aserr: protocol mismatch or lose features such as new commands. Until all nodes run the same binary, it is safest to keep the cluster on the older version or pause traffic that depends on the new features.

Quick Verification Checklist

Check	Command	Expected Result
Replication factor	asinfo -v replication-factor	>1 for safe continuity
Protocol version	asinfo -v protocol-version	Same across all nodes
Node health	asinfo -v node-status	Online
Cluster size	asinfo -v cluster-size	Equal to expected node count

Missing Diagnostic Detail Needed

To give a definitive recommendation, I need to know the current replication factor for your namespaces and the protocol version of the failed node versus the healthy nodes. Once those are confirmed, I can advise whether to pause traffic or proceed.