Troubleshooting K3s HA Control-Plane Membership and etcd Failures

The Problem: Control-Plane Join Failures

When scaling a K3s High Availability (HA) cluster using embedded etcd, adding a new control-plane node often results in the node remaining in a NotReady state. This is typically accompanied by etcdserver: request timed out or raft: bad membership change errors in the system logs. These failures prevent the cluster from reaching a healthy quorum, leaving the cluster vulnerable to total outage if a remaining node fails.

Diagnostic Matrix: Symptoms and Root Causes

Use this table to map the observed behavior to the likely cause and the primary diagnostic command.

Symptom	Likely Cause	Diagnostic Command
New node stuck at "Starting k3s"	Cluster token mismatch	cat /var/lib/rancher/k3s/server/token
raft: bad membership change	Concurrent join attempts	kubectl -n kube-system get cm k3s-etcd
Leader flapping / unstable quorum	System clock skew	chronyc tracking
etcd CrashLoopBackOff	Disk I/O latency (fsync)	fio --name=test --rw=write --fsync=1

Step-by-Step Membership Verification

Follow these checks in order to isolate the failure point. These instructions assume K3s v1.28 through v1.30.

1. Token Consistency: Verify the cluster token is identical across all existing and joining control-plane nodes. The token is read at first start; if it was changed on the seed node after the cluster was initialized, new nodes will fail to authenticate.
2. Network Reachability: Ensure the following TCP ports are open between all control-plane IP addresses:
   • 6443: Kubernetes API Server
   • 2379: etcd client communication
   • 2380: etcd peer-to-peer communication
3. Clock Synchronization: Check that the time offset between nodes is less than 50ms. etcd relies on precise timing for leader election and lease management. Use timedatectl status or chronyc tracking.
4. etcd Membership State: Inspect the current raft membership. Run this on the current leader node:

   # Run as root or with sudo
   /var/lib/rancher/k3s/data/bin/etcdctl member list

   Look for duplicate node names or IDs that do not match the current physical hardware.
5. Disk Performance: Review the fsync latency on the directory /var/lib/rancher/k3s/server/db. etcd is highly sensitive to disk write latency; slow disks cause heartbeat timeouts and leader flapping.

Remediation Paths

Apply the fix corresponding to the finding from the steps above.

Finding: Token Mismatch

Copy the token from the first initialized node to /var/lib/rancher/k3s/server/token on the failing node. Since the token is only read at first start, you must wipe the local database before restarting:

# Run on the failing node
rm -rf /var/lib/rancher/k3s/server/db
systemctl restart k3s

Finding: Stale or Corrupt Membership

If etcdctl member list shows a node that no longer exists or a failed join attempt, remove the stale member from the leader node:

# Run on the current cluster leader
/var/lib/rancher/k3s/data/bin/etcdctl member remove <MEMBER_ID>

Caution: Never run member remove on a learner node; only target voting members that are confirmed offline.

Finding: High Disk Latency

If the underlying storage cannot sustain the etcd write load, migrate the data directory to an SSD or use an external etcd cluster by adding the --datastore-endpoint flag to the K3s server configuration.

Verification of Recovery

To confirm the cluster is healthy and the new node is fully integrated, perform these three checks:

1. Node Status: Run kubectl get nodes -o wide. All control-plane nodes must show Ready.
2. etcd Health: Run the following to ensure all endpoints are responding and synchronized:

   kubectl -n kube-system exec etcd-<node-name> -- etcdctl endpoint health --cluster

3. Scheduling Test: Deploy a test workload with 3 replicas. Verify that pods are scheduled across different control-plane nodes to ensure the API server is functioning across the quorum.

Escalation Criteria

If the following conditions persist after the above fixes, escalate to a senior infrastructure engineer or Rancher support:

• Quorum Loss: More than one node is down in a 3-node cluster, preventing a majority vote.
• Persistent Latency: Disk fsync latency remains above 100ms after migrating to SSDs.
• Version Skew: Control-plane nodes differ by more than one minor version (e.g., mixing v1.28 and v1.30).
• Recurring Corruption: Membership corruption returns immediately after a clean re-join.