YugabyteDB Synchronous Multi-AZ Architecture: Requirements, Minimal Design, and Operational Boundaries

The Problem: Zero Data Loss Across Availability Zones

Teams adopting YugabyteDB for geo-distributed OLTP often need to survive a full availability zone (AZ) outage with RPO=0 (zero data loss) and RTO<5s failover. The database achieves this through Raft consensus with replication factor 3 (RF=3) across three AZs in a single region. This note documents the smallest viable deployment, where trust boundaries lie, what to monitor, and when the design must change.

Requirements Driving the Design

• Survive single-AZ loss with no committed data lost and automatic leader failover under 5 seconds.
• OLTP latency targets: write p99 < 10ms, read p99 < 5ms within the region.
• Compliance: encryption in transit (TLS 1.2+) and at rest (AES-256) enforced cluster-wide.
• Operational simplicity: minimal node count, standard tooling for verification.

These requirements map directly to a 3-node cluster (one per AZ) with RF=3. Each node runs both yb-master and yb-tserver. Tablet leaders distribute evenly across the three AZs; client drivers (Java, Go, Node) use topology-aware load balancing and can fall back to follower reads for read scaling.

Smallest Suitable Design: 3 Nodes, 3 AZs, RF=3

Deploy one node in each AZ (e.g., us-east-1a, us-east-1b, us-east-1c). Start each with:

yugabyted start \
  --cloud_location.az=us-east-1a \
  --master_addresses=node1:7100,node2:7100,node3:7100 \
  --tserver_flags="placement_cloud=aws,placement_region=us-east-1,placement_zone=us-east-1a" \
  --enable_tls=true \
  --certs_dir=/mnt/yugabyte/certs

Where to run: on each VM/instance as the yugabyte user (or root for systemd). Permissions: write access to --base_dir (default ~/yugabyte-data) and certs directory. Placeholders: replace AZ names, node hostnames, and cert paths.

Verify leader distribution after startup:

yb-admin -master_addresses node1:7100,node2:7100,node3:7100 list_tablets \
  | awk '/LEADER/ {print $NF}' | sort | uniq -c

Expect roughly equal leader counts per AZ. A skew >10% triggers a manual rebalance (see Operational Checks).

Trust and Data Boundaries

Each AZ is a fault domain. Raft quorum (2 of 3) crosses AZ boundaries — no single AZ holds authority. This means:

• A write commits only after replication to a second AZ.
• If one AZ loses network or power, the remaining two form quorum and continue serving writes.
• The isolated AZ's tablet leaders step down via Raft pre-vote (enabled by default in 2.20+).

Encryption is enforced cluster-wide: TLS 1.2+ for all RPC and client connections; AES-256 at rest via --encryption_at_rest flag. For disaster recovery, xCluster replication uses a separate trust domain with its own certificate authority — do not reuse the primary cluster's certs.

Operational Checks: What to Monitor and Alert On

Metric	Source	Alert Threshold	Action
Leader skew per AZ	yb-admin master_leader_status, tablet_leader_distribution	>10% deviation from 33%	Run yb-admin change_table_placement or wait for auto-rebalance
Raft replication lag	yb-tserver /metrics → raft_replication_lag_seconds	>500ms sustained	Investigate disk/network on lagging peer; consider follower read staleness impact
Write latency p99	handler_latency_yb_tserver_TabletServerService_Write_p99	>10ms	Check cross-AZ RTT; if >2ms, latency budget at risk
Tablet count per tserver	tablet_count_per_tserver	Skew >20%	Trigger manual tablet split/rebalance
HLC clock skew	hybrid_clock_skew_max metric	>500ms	Fix NTP/Chrony; writes stall until sync

Run these checks from a bastion host with network access to master/tserver RPC ports (7100, 9100). Requires yb-admin binary (included in YugabyteDB distribution).

Failure Modes and Expected Behavior

1. Network Partition Isolating One AZ

The minority partition (1 node) loses quorum. Its tablet leaders step down via Raft pre-vote; the majority partition (2 nodes) continues serving writes with a new leader elected in the remaining AZs. Client drivers with topology-aware load balancing redirect traffic automatically.

Verification: inject partition with tc qdisc add dev eth0 root netem loss 100% on one node's interface. Confirm minority steps down in yb-tserver logs ("Stepping down as leader") and majority continues accepting writes via yb-loadtester.

2. Slow Disk in One AZ

Tablet leadership migrates away from the slow node via Raft pre-vote. The affected tablet's replicas in healthy AZs elect a new leader. Monitor raft_replication_lag_seconds on the slow peer — it will spike before leadership transfer.

3. Clock Skew >500ms

YugabyteDB uses Hybrid Logical Clocks (HLC). If NTP drifts beyond 500ms, HLC bounds are violated and writes stall until clocks resynchronize. Logs show "HLC violation: clock skew too large". Fix: ensure chronyd or ntpd runs with local reference clocks; avoid date -s manual adjustments.

Verification: simulate with date -s '+10 seconds' on one node (requires root). Observe write stalls and HLC errors in logs. Risk: this test disrupts the cluster; run only in staging.

Conditions That Change the Design

Trigger	Design Change	Trade-off
Multi-region latency >5ms	Switch to async xCluster replication	RPO >0 (seconds to minutes); no synchronous commit across regions
Read-heavy workload needing local reads	Enable follower reads with staleness bound (--follower_read_staleness_ms=30000)	Stale reads up to replication lag; not read-your-writes consistent without app coordination
Compliance: data residency per AZ	Pin tablets via placement policies (yb-admin modify_placement_info)	Reduces fault tolerance if pinned AZ fails; quorum may not form
Add 4th AZ	Increase RF to 4 or adjust placement policy	Quorum becomes 3/4; write latency increases (extra network hop)

Master Leader Placement Constraint

The yb-master leader must reside in an AZ that can form quorum. If the master leader's AZ loses quorum (e.g., network partition), cluster metadata operations (DDL, tablet splits, placement changes) pause until a new master election completes in the majority partition. This does not affect data reads/writes — only control-plane operations.

Verify master leader location:

yb-admin -master_addresses node1:7100,node2:7100,node3:7100 master_leader_status

Limitations and Verification Checklist

• Cross-AZ latency >2ms significantly increases write p99. Benchmark with actual network (e.g., yb-loadtester --write_only --num_threads 32 --duration 300) before production.
• Follower reads return data up to replication lag stale. Do not use for read-your-writes consistency without application-level coordination (e.g., read-after-write verification).
• Tablet splitting under load can temporarily unbalance leaders. Monitor tablet_count_per_tserver and trigger manual rebalance if skew >20%.
• Adding a 4th AZ requires RF=4 or placement policy change; quorum becomes 3/4, increasing write latency.

Pre-production verification steps (run in staging mirroring production topology):

1. Deploy 3-node cluster across 3 AZs using yugabyted start --cloud_location.az=....
2. Verify placement with yb-admin get_placement_info — confirm replicas match intended AZ distribution.
3. Run yb-loadtester for 5 minutes; capture p99 latency and replication lag from /prometheus-metrics.
4. Inject network partition on one AZ; confirm minority steps down, majority continues.
5. Simulate clock skew; observe HLC violations and write stalls.
6. Review YugabyteDB 2.20+ release notes for Raft pre-vote, follower read staleness metrics, and placement policy enhancements.

This architecture holds while requirements stay within single-region, sub-5ms AZ latency, and RPO=0. The moment any constraint shifts — multi-region, relaxed RPO, data residency — the design changes as documented above.