Raft consensus failure preventing leader election in Consul cluster

Question

When a Consul cluster fails to establish a leader, the agent logs often indicate that the Raft protocol cannot achieve a majority. This behavior typically occurs in environments where nodes are reachable but remain in a follower or candidate state indefinitely, despite valid connectivity between the configured cluster members.

The challenge lies in distinguishing between transient network latency and a permanent failure caused by stale state files or mismatched bootstrap expectations. While the logs show repeated attempts to reach a quorum, they do not always explicitly state whether the failure is due to a specific network partition or a configuration mismatch in the join-list.

What specific log patterns differentiate a network-induced partition from a configuration-based failure to reach a quorum? How can one determine if a node is failing to join due to an outdated local raft index versus a mismatch in the expected cluster size?

Tasadduq Burney · Answer

Identifying the Root Cause of Quorum Failure
To differentiate between network partitions and configuration mismatches in a Consul cluster, you must analyze the specific failure patterns in the agent logs and compare the Raft state across nodes.

Log Pattern Differentiation
The following patterns typically distinguish the two failure modes:

Network-Induced Partitions: Look for explicit connectivity errors. Common indicators include "Failed to connect to peer", "heartbeat timeout", or "peer unreachable". These logs suggest the Raft protocol is attempting to communicate but the transport layer is failing.
  Configuration-Based Failures: These are characterized by a lack of connectivity errors but a failure to reach a consensus. Look for "Not enough peers to form a quorum" or "Failed to load cluster configuration". This indicates the nodes can see each other, but they do not agree on who belongs in the cluster or how many votes are required for a leader.

Stale Raft Index vs. Cluster Size Mismatch
To determine if a node is failing due to an outdated local index or a size mismatch, use the following verification steps:

Check Cluster Size: Run consul members on each node. Compare the reported number of members against your expected configuration. If a node reports a different expected cluster size (e.g., expecting 5 nodes when only 3 are deployed), you have a configuration mismatch.
  Verify Raft Index: Run consul operator raft status. Compare the commit index of the failing node against the leader's index. If the node's index is significantly lower and the logs show "Raft log is behind" or "Failed to apply log entry", the node is suffering from a stale state.

Resolution Steps
Depending on the finding, apply the corresponding fix:

For Stale State: Stop the Consul agent, clear the data_dir (specifically the Raft logs), and restart the agent to allow it to perform a fresh snapshot sync from the leader.
  For Configuration Mismatch: Verify the retry_join or bootstrap_expect settings in your configuration files to ensure all nodes agree on the initial cluster size.

Missing Diagnostic: Please provide the output of consul operator raft status from at least two nodes; this is required to confirm if the nodes are in the same term or if a term mismatch is preventing the election.

Raft consensus failure preventing leader election in Consul cluster

1 answer

Identifying the Root Cause of Quorum Failure

Log Pattern Differentiation

Stale Raft Index vs. Cluster Size Mismatch

Resolution Steps

Your perspective belongs here.

0 question comments