Troubleshooting Cassandra Node Join Failures in Kubernetes StatefulSet Deployments

When a Cassandra pod in a StatefulSet fails to join the cluster, follow these systematic steps to identify the root cause and restore normal operation. Each step includes verification actions and a rollback option if changes cause further problems. This guidance is based on general Cassandra and Kubernetes practices; review it in your environment before applying.

1. Verify pod and StatefulSet status
   • Run kubectl get statefulset and kubectl get pods -l app=cassandra to confirm the pod is running, not in CrashLoopBackOff, and that the StatefulSet reports the expected replica count.
   • If the pod is pending or failing, check events with kubectl describe pod <pod-name> for scheduling or volume‑mount issues.
2. Rollback: If a recent StatefulSet update caused the problem, roll back to the previous revision using kubectl rollback undo statefulset/cassandra.

1. Inspect Cassandra logs
   • Retrieve logs with kubectl logs <pod-name> --container cassandra (adjust container name if needed). Look for messages such as "Failed to join", "Unable to connect to seed", "Timeout", or authentication errors.
   • Note any stack traces or configuration warnings that indicate missing seeds, incorrect listen/rpc addresses, or schema version mismatches.
2. Rollback: If logs reveal a bad configuration change, revert the ConfigMap or Secret that holds cassandra.yaml to its last known good version and trigger a pod restart.

1. Validate network and DNS resolution
   • From a temporary busybox pod in the same namespace, execute nslookup <seed-service> to ensure the Cassandra service name resolves to the expected cluster IPs.
   • Test connectivity to each seed node on the Cassandra ports (typically 7000 for intra‑cluster and 9042 for client) using nc -zv <seed-ip> 7000.
   • Confirm that the pod’s POD_IP matches the address advertised in cassandra.yaml (listen_address, rpc_address, broadcast_address).
2. Rollback: If a NetworkPolicy or Service definition was altered incorrectly, restore the previous version and re‑apply.

1. Check Cassandra configuration consistency
   • Compare the cassandra.yaml (or equivalent ConfigMap) used by the failing pod with that of a healthy node. Pay attention to cluster_name, seeds, endpoint_snitch, authenticator, and authorizer settings.
   • Ensure that the seeds list includes at least one reachable node and that the seed nodes themselves are up and reporting normally (nodetool status).
2. Rollback: If a mismatched configuration is detected, replace the ConfigMap with the version from a healthy pod and delete the problematic pod to let the StatefulSet recreate it.

1. Verify storage and persistence
   • Check that the PersistentVolumeClaim bound to the pod is healthy (kubectl get pvc) and that the underlying volume has no I/O errors (review node logs or cloud provider volume status).
   • Ensure the data_file_directories path in cassandra.yaml points to the mounted volume and that the Cassandra user has write permissions.
2. Rollback: If the volume is corrupted, detach it, restore from a recent snapshot or backup, and re‑attach before recreating the pod.

1. Confirm node has joined the ring
   • Once the pod is running, execute kubectl exec -it <pod-name> -- nodetool status (or nodetool info) from inside the container. The node should appear as "Up Normal" (UN) or at least "Joining" transitioning to UN.
   • If the node stays in "Joining" or "Leaving", repeat steps 2‑4, focusing on gossip timeout or schema disagreement messages.
2. Rollback: If the node cannot join after exhausting the above, delete the pod (kubectl delete pod <pod-name>) to allow the StatefulSet to start a fresh instance, which will re‑attempt the join process with the current configuration.

Throughout the process, document each change and its outcome. If the issue persists after verifying configuration, network, and storage, consider deeper causes such as incompatible Cassandra versions across pods, JVM memory limits, or insufficient CPU/resources, and adjust the StatefulSet resource requests/limits accordingly.