YugabyteDB Cross-Tablet Transactions: Raft, HLC, and 2PC Design Boundaries

The problem: atomic writes across independent tablets

YugabyteDB stores data in tablets—horizontal partitions spread across nodes. When a single transaction updates rows in two different tablets, it cannot rely on one local disk commit. The database must coordinate writes across independent Raft consensus groups while still offering readers a consistent point-in-time view. The practical takeaway: before tuning queries, confirm where each tablet's leaseholder sits, how the hybrid logical clock orders events, and what a two-phase commit costs when it crosses a shard boundary.

Requirements for the transaction layer

The design must satisfy four constraints simultaneously:

• Atomicity: All writes in a transaction commit, or none do, even when tablets live on different nodes.
• Causal ordering: Events must be ordered consistently across nodes without depending on perfectly synchronized atomic clocks.
• Availability under failure: A minority of node failures must not block progress, provided a Raft quorum remains.
• Non-blocking reads: Readers should see a stable snapshot without impeding concurrent writers.

The smallest suitable design: three nodes, tablets, and Raft

The smallest topology that tolerates one failure while preserving quorum is three nodes. Data is divided into tablets; each tablet is replicated by a Raft group. Within each group, one replica acts as the leaseholder—the temporary leader that owns writes and authoritative reads for that tablet. Followers replicate the log but do not serve the same authoritative role unless explicitly configured.

For ordering, YugabyteDB uses a hybrid logical clock (HLC). An HLC combines a physical wall-clock component with a logical counter, giving nodes a monotonically increasing timestamp that respects causality even when physical clocks drift slightly. This avoids the need for atomic clocks or a global time service.

When a transaction touches a single tablet, the leaseholder can often process it locally. When it spans multiple tablets, a transaction manager coordinates a two-phase commit (2PC) variant: participants prepare, a decision is recorded, and then commits are applied. The transaction status is itself stored in a special tablet so that recovery is possible.

Trust and data boundaries: who may read and write

The leaseholder is the authority for its tablet. Clients should send writes and strongly consistent reads to the current leaseholder. Followers hold replicated data, but treating them as equivalent to the leaseholder breaks the consistency model unless follower reads are explicitly enabled and the leaseholder confirms the follower's snapshot is current.

Read-only transactions use snapshot isolation: a transaction observes the database as of a single HLC timestamp and does not block concurrent writes. The boundary is the tablet. Single-tablet operations stay inside one trust domain. Cross-tablet operations cross domains and inherit the latency and failure complexity of 2PC.

Operational checks: inspecting tablets and leaseholders

Run these commands from an administration host with network access to the master RPC endpoint. Replace placeholders with your deployment values.

# List all tablets and their Raft configurations
yb-admin --master_addresses <master-addresses:7100> list_tablets

# Show the leaseholder and follower locations for a specific tablet
yb-admin --master_addresses <master-addresses:7100> get_tablet_placement <tablet-id>

The yb-admin commands require network reachability to the YB-Master RPC endpoint. They are read-only metadata operations, but running them against an incorrect address will return connection errors rather than cluster state. Expected checks: list_tablets should return tablet IDs with a Raft configuration showing three voters. get_tablet_placement should identify one leader UUID and two follower UUIDs. If a tablet shows no leader, Raft election is in progress.

To inspect the cluster visually, open the YB-Master and YB-TServer web interfaces (default ports 7000 and 9000; verify against your deployment). The tablets page shows leaseholder distribution across nodes.

For a behavioral check, connect with ysqlsh -h <host> -U <user> -d <database> and run an explicit transaction that updates rows in two tables known to reside on different tablets. This requires valid database credentials. Observe whether the commit succeeds immediately or retries. Do not assume a specific latency; compare it against a single-tablet update in the same session.

Failure modes and diagnostic decisions

Condition	Diagnostic signal	Decision
Clock skew beyond configured maximum offset	HLC cannot guarantee safe ordering; transactions may restart or degrade	Verify NTP or equivalent time synchronization on all nodes before blaming the database
Hotspotting on a single tablet	One leaseholder shows disproportionate CPU, Raft log append rate, or latency	Re-evaluate the primary key or sharding strategy; add tablets or split ranges
Cross-shard latency	Multi-tablet transactions consistently slower than single-tablet ones	Co-locate related rows or redesign the schema to reduce cross-tablet writes
Node failure during 2PC	Transaction coordinator cannot reach a participant; commit status may be ambiguous	Allow the transaction manager to recover using the status tablet; avoid manual intervention that could violate atomicity

Conditions that would change the design

If your workload is overwhelmingly single-tablet, the cross-shard 2PC penalty is rare, and the default design is sufficient. If most transactions span shards, consider co-locating related data or using a different isolation level where acceptable.

If stale reads are tolerable for analytics, enabling follower reads can reduce load on leaseholders. If your environment cannot maintain tight clock synchronization, the HLC's tolerance has limits; exceeding the maximum clock offset will force restarts regardless of query tuning.

Finally, if a single tablet becomes a persistent bottleneck, the answer is usually schema or partitioning change, not simply adding nodes. Adding nodes spreads tablets but does not fix a poorly chosen shard key.

Limitations and verification

This note describes the general architecture. Exact flags, default ports, and admin command syntax vary by YugabyteDB version and deployment method. Before relying on any behavior in production, confirm the current defaults in the official documentation for your release. A practical verification is to deploy a three-node cluster, run the yb-admin checks above, execute a cross-tablet transaction, and confirm that leaseholders remain distributed and that transaction retries stay within your application's tolerance.