Limits on Failover Latency in YugabyteDB Raft Leader Election

Answer

YugabyteDB does not provide a hard, system‑wide upper bound on the time to elect a new tablet leader after a node failure. The configured raft_election_timeout_ms_max sets a baseline for the worst‑case election timeout, but the actual failover latency also includes the time needed to gather votes from a majority of replicas, which depends on network latency, replica responsiveness, GC pauses, and cluster size. Consequently, the observed interval between an Election timeout log entry and the corresponding Leader elected entry can exceed the configured maximum.

Confirmed Facts

• The election timeout for each Raft group is chosen randomly from the interval [raft_election_timeout_ms_min, raft_election_timeout_ms_max].
• Logs from yb‑master and yb‑tserver contain messages like Election timeout (when a follower’s timer expires) and Leader elected (when a candidate secures a majority).
• YugabyteDB documentation does not publish a hard upper bound beyond the configured max timeout; the only guarantee is that a node will not start a new election before its timeout expires.
• Default timeout values differ between major releases: 2.x uses smaller defaults (e.g., 500 ms min, 1500 ms max) while 3.x uses larger defaults (e.g., 5000 ms min, 10000 ms max).
• Cluster size (replication factor) influences the majority size; larger clusters may require more votes, potentially increasing the vote‑gathering phase under high latency.

Likely Explanation

The random election timeout introduces a probability distribution for the start of a new election. After the timeout expires, a candidate must receive votes from a majority of live replicas; this vote‑gathering phase adds variable latency that is not bounded by the timeout configuration. Network partitions, GC pauses, or slow replicas can dominate the total failover time, making the configured max timeout an insufficient guarantee for SLA planning.

Steps to Observe and Measure Failover Latency

1. Determine the current election timeout settings:

   yb-ctl --timeout 10s get_flag raft_election_timeout_ms_min
    yb-ctl --timeout 10s get_flag raft_election_timeout_ms_max
   

   (or check the Admin UI → Flags tab).
2. Induce a controlled failure (e.g., stop a tablet server):

   yb-ctl stop_node <node-id>
   

3. Capture the relevant logs around the failure. For each tablet, locate the pair of entries:

   grep -E "Election timeout|Leader elected" yb-tserver.INFO | grep <tablet-id>
   

4. Extract the timestamps (wall‑clock) of the Election timeout and the subsequent Leader elected for the same tablet and compute the delta.
5. Repeat the failure test multiple times (e.g., 10–20 runs) to build an empirical distribution of the observed delta.
6. If a tighter bound is required, measure the one‑way network RTT between replicas and the typical GC pause duration; add these to the configured max timeout as a conservative estimate.

Missing Diagnostic Detail

To refine the recommendation, it would be helpful to know the observed network latency or GC pause duration during your failover tests, as these factors directly affect the vote‑gathering phase and can change whether the configured max timeout alone is sufficient for your SLA.