Diagnosing Intermittent Query Failures from Leader Loss in YugabyteDB

Recognizable Condition

Applications experience sporadic query timeouts, connection resets, or "leader not found" errors that resolve on retry. These symptoms appear randomly across tablets and often correlate with brief network blips or tablet server restarts. The cluster reports healthy overall, but individual requests fail with errors like ERROR: Tablet leader not found or client-side socket timeouts.

Cause and Diagnostic Quick Reference

Observed Symptom	Likely Root Cause	Primary Diagnostic Command
Intermittent "leader not found" or request timeout	Leader election in progress due to network partition, slow heartbeats, or tablet server crash	yb-ctl get-leader <table_id>.<tablet_id>
Client sees connection reset after successful queries	Tablet server hosting leader restarted; client cache stale	yb-ctl list (check server status)
Elevated latency on specific tablets only	Leader on overloaded node; heartbeat delays trigger re-election	yb-ctl cluster-status + tablet server metrics
Failover takes longer than expected (seconds)	Quorum loss or insufficient replicas for replication factor	Verify replication factor vs. live nodes: yb-admin -master_addresses <masters> list_tablets

Ordered Diagnostic Checks

Run these steps in sequence from a bastion host or admin workstation with yb-ctl and yb-admin in PATH. Requires cluster admin credentials (typically the yugabyte OS user or equivalent RBAC role).

1. Cluster Health Overview

yb-ctl cluster-status --master_addresses <master1:7100,master2:7100,master3:7100>

Look for: any tablet server in STOPPED or UNREACHABLE state; replication factor mismatches; tablets with LEADERLESS status. Expected: all masters and tablet servers RUNNING, tablet count matches schema.

2. Identify Affected Tablet and Current Leader

# Find tablet ID from application error or slow query log
yb-ctl get-leader <table_id>.<tablet_id> --master_addresses <master_addrs>

If command returns no leader or times out, election is stalled. Note the tablet ID for log correlation.

3. Tablet Server Logs for Election Events

On each tablet server (SSH required, typically /home/yugabyte/tserver/logs):

grep -E "(LeaderElection|StartElection|NewLeader|Heartbeat)" yb-tserver.INFO* | tail -50

Look for: frequent StartElection without NewLeader, heartbeat timeouts (Heartbeat missed), or TabletServer::Shutdown entries. Correlate timestamps with application errors.

4. Network Connectivity Between Nodes

# From each tablet server, test TCP to peers on RPC ports (default 9100, 11000)
for host in <peer1> <peer2> <peer3>; do
  nc -zv $host 9100 11000 2>&1 | grep -E "succeeded|failed"
done

Also check ICMP latency: ping -c 5 <peer>. Expect < 1 ms intra-AZ, < 5 ms cross-AZ. Persistent > 10 ms or packet loss triggers election instability.

5. Resource Saturation on Tablet Servers

# Check CPU, disk I/O, memory on each tablet server
top -bn1 | head -20
iostat -xz 1 3
free -h

High CPU (> 80% sustained), disk utilization > 80%, or OOM kills cause heartbeat delays. YugabyteDB 2.18+ exposes Prometheus metrics at :9000/metrics; query process_cpu_seconds_total and disk_write_bytes_total for trends.

Fixes Tied to Diagnostic Findings

Finding: Single Tablet Server Unreachable or Crashed

Action: Restart the affected tablet server process.

# On the problematic node (requires sudo or yugabyte user)
systemctl restart yb-tserver
# Or if using yb-ctl managed cluster:
yb-ctl restart_tserver <tserver_id>

Risk: Brief unavailability for tablets hosted on that node. Ensure replication factor >= 3 so quorum survives one node loss. Verify recovery with yb-ctl cluster-status within 30 seconds.

Finding: Heartbeat Timeouts Due to Network Latency

Action: Increase Raft heartbeat interval. Default is 500 ms; raise to 1-2 seconds for cross-region deployments.

# On all masters (requires master restart, schedule maintenance window)
# Add to master gflags or yb-master.conf:
--raft_heartbeat_interval_ms=1500
--raft_election_timeout_ms=5000

Verification: After rolling master restart, confirm new values via curl <master>:7000/varz | grep raft_heartbeat. Monitor election frequency for 24 hours.

Finding: Leader on Overloaded Node (High CPU/Disk)

Action: Rebalance tablets or add capacity.

# Option A: Trigger manual leader rebalance for specific table
yb-admin -master_addresses <masters> change_table_config <keyspace>.<table>> 0 0 LEADER_BALANCE

# Option B: Add new tablet server and let auto-rebalance distribute load
yb-ctl add_node --master_addresses <masters>

Limitation: LEADER_BALANCE moves leaders only; data rebalancing requires yb-admin change_table_config ... REBALANCE which streams data and increases I/O temporarily.

Finding: Quorum Loss from Insufficient Live Replicas

Action: Verify replication factor matches node count; replace failed nodes promptly.

# Check tablet replica placement
yb-admin -master_addresses <masters> list_tablets <table_id> | grep -A5 <tablet_id>

If replicas < replication factor, add node or wait for failed node recovery. Do not force leader change (yb-admin change_table_config ... FORCE_LEADER) without confirming quorum; this risks split-brain.

Finding: Known Election Bug in Current Version

Action: Apply latest patch release. YugabyteDB 2.18.3+, 2.19.1+, and 2.20.0+ contain fixes for election storms under network partition (see release notes for YB-XXXXX tickets). Upgrade during maintenance window using rolling upgrade procedure.

Escalation Criteria

Engage YugabyteDB support or senior platform engineer when:

• Elections recur > 5 times/hour across multiple tablets after applying above fixes
• Cluster-wide latency remains > 2x baseline for > 15 minutes post-fix
• Master leadership changes frequently (check yb-ctl get-leader for master itself)
• Data inconsistency suspected: row counts differ between replicas (run yb-admin verify_tablet)

Verification After Fix

1. Run yb-ctl cluster-status --master_addresses <masters> — all servers RUNNING, no LEADERLESS tablets.
2. Execute representative workload (e.g., pgbench -c 10 -T 600 against affected tables) and confirm:
   • Zero connection errors for 10+ minutes
   • P99 latency within 20% of pre-incident baseline
   • No "leader not found" in application logs
3. Monitor Prometheus yb_tablet_leader_election_total — should remain flat.

Limitations and Notes

• This guide assumes YugabyteDB 2.18+ with default Raft configuration. Behavior may differ in 2.14 or earlier.
• Cloud-managed (YugabyteDB Managed) users lack yb-ctl access; use cloud console metrics and support tickets instead.
• Network-level issues (security groups, VPC peering, CNI) require infrastructure team involvement beyond database diagnostics.
• No rollback procedure included: configuration changes (gflags) are reversible via rolling restart with previous values; node addition is irreversible without decommission.