Using YugabyteDB Read Replicas with Timeline Consistency for Geo‑Distributed Reads

Problem: reads are slow for users far from the primary region

Your application writes to a single YugabyteDB primary region (e.g., us-east) but serves read‑only traffic from users in us-west and eu-west. Because every read must go to the primary leader, latency adds up and the primary becomes a bottleneck. You need a way to serve reads locally while keeping the data fresh enough for your business logic.

Takeaway

Configure YugabyteDB read replicas with timeline consistency (bounded staleness). Reads can be served from the nearest replica region with a configurable max_staleness_ms bound, and you can monitor replication lag to know when the bound is violated.

How timeline consistency works

Each tablet has a leader that stays in the primary region. Follower tablets in replica regions receive the same WAL entries asynchronously. A follower maintains a safe_time – the latest hybrid_time it knows is committed on the leader. When a client opens a read‑only transaction and specifies max_staleness_ms, the replica delays the read until its safe_time is newer than the requested read time (current hybrid_time minus the staleness bound). This guarantees that the data returned is no older than the bound without requiring a quorum read.

Setting up a 3‑region replica cluster

1. Install YugabyteDB 2.18+ (the version that introduced timeline consistency).

2. On a machine with network access to all three regions, start a local test cluster with placement info that defines one primary region and two replica regions. Example using yb-ctl (run as a user that can execute the binary):

   yb-ctl create --replication_factor 3 \
       --placement_info "us-east:1,us-west:1,eu-west:1" \
       --num_nodes 3
   

   This command creates three nodes, each assigned to a region via the placement flag. The leader for each tablet will remain in us-east; followers will be placed in us-west and eu-west.

3. Verify tablet distribution:

   yb-admin list_tablets

   Look for the leader field showing us-east for all tablets and followers showing the replica regions.

Routing reads and handling partitions

Application side:

• Use a load‑balancer or driver‑aware routing (YSQL/JDBC/pgx) that directs BEGIN TRANSACTION READ ONLY to the nearest replica region.

• Write transactions (BEGIN without READ ONLY) must go to the primary region; most drivers allow you to specify different host lists for reads vs. writes.

• When opening a read‑only transaction, set the staleness bound:

  SET TRANSACTION READ ONLY;
  SET LOCAL yb.max_staleness_ms = '5000';  -- 5 seconds
  SELECT * FROM orders WHERE customer_id = 123;
  COMMIT;

  The replica will block the SELECT until its safe_time advances enough that the data is no older than 5 seconds. If the network between primary and replica pauses, safe_time stops advancing; after the staleness window expires the read returns an error (or times out, depending on client configuration) rather than returning unbounded stale data.

Trade‑offs and operational checklist

• Consistency model: Timeline consistency is not linearizable. Two concurrent writes may appear in different orders on different replicas if they fall within the staleness window. Use this pattern only for workloads that can tolerate bounded staleness (e.g., product catalogs, analytics dashboards).
• Cost: Each replica stores a full copy of the data, so storage and CPU scale linearly with the replication factor and number of replica regions.
• DDL impact: Schema changes propagate asynchronously. While a replica is applying an ALTER TABLE or creating an index, it may temporarily reject reads on the affected table. Plan DDL during low‑traffic windows or use online schema migration tools.
• Failure handling: Automatic failover is not built‑in. To promote a replica region to primary you must run yb-admin change_replica_placement manually (or via orchestration). Document the runbook and test it regularly.
• Monitoring: Key metrics to watch (expose via Prometheus):
  • yb_tablet_replication_lag_ms – per‑tablet WAL lag.
  • yb_safe_time_lag_ms – difference between cluster hybrid_time and replica safe_time.
  • yb_read_replica_staleness_ms – observed staleness for read‑only transactions.
  Set alerts if any lag exceeds your chosen max_staleness_ms for more than a minute.

Practical verification

After applying the configuration, you can verify that the staleness bound is respected:

1. Run a write workload in the primary region (e.g., pgbench inserting rows).
2. From a client in a replica region, execute a read‑only transaction with SET LOCAL yb.max_staleness_ms = '2000' and record the latency.
3. Check the metric yb_safe_time_lag_ms for the replica tablet; it should stay below 2000 ms while the partition is healthy.
4. Introduce a network delay or partition between primary and replica (using tc or a chaos tool). Observe that read latency rises and eventually the read returns an error after the staleness window expires. When the partition heals, reads resume and lag drops back below the threshold.

These steps give you confidence that the bound is working as expected without claiming any specific test results.

Actionable closing

If your application suffers from cross‑region read latency, start by deploying a read replica in each user‑proximate region with a replication factor of 3 and placement info that keeps leaders in your primary zone. Configure your data‑access layer to open read‑only transactions with a realistic max_staleness_ms (typically 2–5 × expected round‑trip time). Monitor the safe‑time lag and replication lag metrics, and run a failover drill at least quarterly. This approach gives you low‑latency reads while keeping data freshness within a known, controllable bound.