Diagnosing k3s HA Failures with External etcd: Connectivity, TLS, and Quorum Issues

Recognizing the Symptom

When a k3s cluster is configured for high‑availability (HA) with an external etcd data store, the most common failure mode is that the k3s-server process exits with logs such as:

failed to connect to etcd: connection refused
etcd client error: dial tcp: no such host

In the kube-system namespace you’ll see the kube-apiserver and etcd pods stuck in CrashLoopBackOff or Pending state. This guide walks through a systematic diagnosis and remediation path.

Short Cause/Diagnostic Table

Issue	Diagnostic Clue	Likely Cause	Immediate Check
Endpoint Unreachable	"connection refused" or "no such host" in logs	Wrong IP/hostname, DNS mis‑resolution, firewall block	Ping or curl the etcd host from the k3s‑server node
TLS Mismatch	"certificate has expired" or "untrusted CA" errors	Expired or mismatched client/server certs	Inspect cert validity and CN with openssl
Quorum Loss	etcd cluster reports insufficient quorum	Peer offline, different cluster ID, network partition	Query etcd members list
Configuration Error	k3s logs show invalid etcd config	Malformed --datastore-endpoint string	Validate k3s config file syntax
Firewall/Policy	Connection attempts time out	Outbound rules block port 2379/2380	Check iptables/ufw rules on all nodes

Ordered Diagnostic Checks

1. Verify etcd endpoint reachability
   

   Run on the host running k3s-server:

   curl -k https://:2379/v2/members
   

   Expect a JSON payload listing cluster members. If you get a network error, the endpoint is unreachable.

2. Check DNS / /etc/hosts resolution
   

   Ensure the hostname in your --datastore-endpoint matches a resolvable IP:

   nslookup 
   

   If the IP differs from the actual etcd node, update /etc/hosts or your DNS zone.

3. Inspect TLS certificates
   

   On the k3s node, view the client cert used for etcd:

   openssl x509 -in /etc/rancher/k3s/k3s.etcd-client.crt -noout -text | grep -E "Subject|Not After"
   

   Confirm the Common Name matches the etcd server’s CN and that the certificate has not expired. Repeat for the server cert if you have access.

4. Validate etcd cluster ID and quorum
   

   Query the etcd cluster directly:

   curl -k https://:2379/v2/members
   

   Check that all expected peers appear and that the cluster-id matches across nodes. If any peer is missing or reports a different ID, bring that node online or re‑initialize the cluster.

5. Confirm firewall and network policies
   

   On every node, ensure outbound traffic to ports 2379 and 2380 is allowed:

   sudo iptables -L OUTPUT -v -n | grep 2379
   

   Adjust rules or open the ports if they are blocked.

6. Review k3s configuration
   

   Open /etc/rancher/k3s/k3s.yaml (or the file used to start k3s) and confirm the --datastore-endpoint value is correct and URL‑encoded if it contains special characters.

Fixes Tied to Findings

• Endpoint Unreachable – Correct the hostname/IP in --datastore-endpoint, update DNS or /etc/hosts, and restart k3s:

  sudo systemctl restart k3s
  

• TLS Mismatch – Replace expired or mismatched certs in /etc/rancher/k3s/ and reload k3s. Use k3s certs if available, or copy fresh certs from the etcd server.
• Quorum Loss – Bring offline peers back online, or re‑configure the cluster ID to match. If a peer is permanently unreachable, remove it from the etcd cluster using the API and then restart k3s.
• Configuration Error – Edit the k3s config file, correct the endpoint string, and validate with k3s --help before restarting.
• Firewall/Policy – Add rules to allow traffic to ports 2379/2380, e.g. sudo ufw allow out 2379/tcp, then restart k3s.

Escalation Criteria

If after applying the above fixes the cluster still fails to start, consider:

• Contact the etcd administrator to verify cluster health and network connectivity.
• Check for network segmentation between k3s nodes and the etcd cluster (VPC security groups, corporate firewalls).
• Verify that the etcd cluster’s cluster-id matches the one expected by k3s; mismatches prevent any node from joining.
• Ensure that the etcd data store is not in a read‑only state due to disk full or corruption; a snapshot backup is recommended before any destructive action.

Practical Verification Checklist

• journalctl -u k3s -n 50 | grep etcd – No connection errors.
• kubectl get pods -n kube-system -o wide – kube-apiserver and etcd in Running.
• Verify etcd API returns members JSON.
• Confirm TLS cert expiration dates are in the future.
• Run etcdctl member list (with proper credentials) to see all peers.

Limitations and Safety Notes

• Modifying etcd certificates or cluster configuration can cause data loss if the cluster state is not backed up. Take a snapshot before making changes.
• Restarting k3s in a production environment may temporarily take the API server offline; schedule maintenance windows accordingly.
• These steps assume k3s v1.28+; earlier releases use an embedded SQLite store and do not support external etcd.