'Failed to acquire leadership' in YB-Master logs: what retry behavior should operators expect?

Direct answer

A YB-Master that logs Failed to acquire leadership does not exit. It remains running as a follower and the Raft layer continuously retries elections after randomized election timeouts. The retry loop is built into the Raft implementation; there is no separate "give up and die" path for this condition.

What controls the retry policy?

The policy is governed by Raft/consensus timing flags. The most relevant are:

1. raft_heartbeat_interval_ms - how often a follower sends heartbeats.
2. leader_failure_max_missed_heartbeat_periods - number of missed heartbeats before a new election is triggered.
3. Catalog-manager leadership flags that coordinate the master-process election.

Exact flag names and defaults are version-sensitive; verify with yb-master --help or the release-specific flag reference for your deployed version.

Version differences (2.x vs 3.x)

• The core retry model is unchanged: a follower keeps re-electing indefinitely while quorum is unavailable.
• Log wording evolved: 3.x often prints Failed to acquire leadership: not enough votes, while 2.x may emit terser lines.
• Default timeouts and some flag names changed between major releases, so 2.x documentation does not necessarily apply to a 3.x cluster.

Distinguishing quorum loss from network-induced timeouts using logs alone

Quorum loss (symmetric failure)

• All surviving masters repeatedly log Failed to acquire leadership (or equivalent) and no node ever logs a successful leader election.
• Peer-unreachable errors name the same down masters on every node.
• No leader heartbeats appear anywhere in the cluster.

Network partition / latency (asymmetric symptoms)

• The majority partition elects a leader normally and logs heartbeats.
• The isolated minority node(s) log election timeouts, RPC-latency warnings, or connection-refused errors to specific peer addresses.
• Heartbeat/election timeout warnings appear only on the partitioned nodes, not cluster-wide.

Minimal verification steps

1. Tail each master's log and confirm whether any node ever logs a successful leader election (look for became leader or leader lease messages).
2. Cross-reference the configured master_addresses list against actually running, reachable yb-master processes.
3. Check for RPC timeout or connection-refused errors tied to specific peer IPs - those indicate network issues rather than process crashes.
4. Run yb-master --help | grep -E 'raft_heartbeat|leader_failure|catalog_manager' on a live node to capture the exact flag names and current values for your version.

One missing diagnostic that changes the recommendation

If step 1 shows no leader ever elected but step 2 confirms a majority of masters are running and reachable, provide the exact YugabyteDB version string (e.g., 2.20.3.0-b123 or 3.1.4.0-b45). Flag defaults and log phrasing are version-specific; with the version I can tell you whether a known default change or flag rename explains the symptom.