Add a Node to a YugabyteDB RF=3 Universe and Verify Tablet Rebalancing

Desired outcome

Add one tablet server to a running YugabyteDB universe at replication factor 3 (RF=3) so existing table replicas and tablet leadership spread onto the new host while reads and writes continue. YugabyteDB's master load balancer handles replica movement automatically; you do not need to split or move tablets manually.

A universe is YugabyteDB's term for a deployed cluster. A tablet is a shard of a table, and RF=3 means each tablet keeps three copies on different nodes. Raft is the consensus protocol that keeps those copies consistent.

Prerequisites

• Existing universe at RF=3 with at least three healthy nodes.
• New host running the same major YugabyteDB version as the cluster (for example, 2.20.x). Do not mix major versions during a rolling upgrade.
• Network reachability from the new host to all master RPC addresses (default port 7100) and tablet server RPC addresses (default port 9100).
• Access to the master UI (typically http://<master-host>:7000) or the yb-admin CLI.
• Enough disk space on the new node for its share of replicas. The load balancer avoids disks above the --fs_space_threshold_percent limit (default 90%).
• Application connection string points to any node or uses a smart driver that refreshes topology metadata.

Procedure

1. Start the new tablet server

Run on the new host as the YugabyteDB service user (typically yugabyte) with write permission on the data directories. Replace placeholders with actual values.

yb-tserver \
  --tserver_master_addrs <master-1>:7100,<master-2>:7100,<master-3>:7100 \
  --fs_data_dirs /mnt/data0,/mnt/data1 \
  --rpc_bind_addresses <new-node-ip>:9100 \
  --webserver_interface <new-node-ip> \
  --webserver_port 9000

If you use yb-ctl for a local test cluster, the equivalent is yb-ctl add_node --tserver_flags="--fs_data_dirs=/mnt/data0,/mnt/data1".

Risk: starting yb-tserver with a different major version can prevent tablet registration or rebalancing.

2. Wait for node registration

The new tablet server registers with the master within seconds. Confirm it appears as LIVE.

yb-admin -master_addresses <master-1>:7100,<master-2>:7100,<master-3>:7100 \
  list_tablet_servers

Run from any host that can reach the masters. Expected output includes the new node UUID with state LIVE.

3. Allow automatic rebalancing

Keep applications running. The master detects the new node and begins moving tablet replicas. Rebalancing is throttled by default (a few concurrent moves per tablet server) and may take minutes. No manual tablet split or move is required for standard RF=3 tables.

Expected checks

Master UI

1. Open http://<any-master>:7000/tablet-servers.
2. Confirm the new node is listed as LIVE with a tablet count that increases over time.
3. Open a table's tablet view and confirm replica sets include the new node.

CLI verification

Get a table ID, then inspect tablet locations periodically.

yb-admin -master_addresses <master-1>:7100,<master-2>:7100,<master-3>:7100 \
  list_tables

yb-admin -master_addresses <master-1>:7100,<master-2>:7100,<master-3>:7100 \
  get_tablet_locations <table-id>

Replace <table-id> with a UUID from list_tables. The new node UUID should appear in replica sets for multiple tablets within several minutes.

Read/write health check

Run a lightweight transaction during rebalancing. For YSQL:

psql -h <any-node> -p 5433 -d yugabyte -c "INSERT INTO test_kv(k,v) VALUES (gen_random_uuid(), 'rebalance-test'); SELECT count(*) FROM test_kv;"

For YCQL:

cqlsh <any-node> 9042 -e "INSERT INTO test.kv (k, v) VALUES (uuid(), 'rebalance-test'); SELECT count(*) FROM test.kv;"

Operations should complete without connection errors. Latency may rise slightly while tablets move, but writes should not fail.

Recovery options

Rebalancing stalls

• Firewall: verify bidirectional ports 7100, 9100, and 9000 between all nodes.
• Disk space: run df -h /mnt/data* on the new node. Replicas will not land on disks above the space threshold.
• Placement constraints: check master logs (~/logs/master.INFO) for messages that a placement policy excludes the new zone or region. Tables with PLACEMENT_BLOCK or PLACEMENT_MINIMUM policies may not use the new node unless its zone is allowed.
• Throttling: the default concurrent move limit is conservative. Raising --load_balancer_max_concurrent_moves usually requires a master restart; test in staging first.

Node fails before rebalancing completes

RF=3 tolerates one node failure. If the new node crashes before receiving a full replica set, traffic continues from the original nodes. Restart the node; it re-registers and rebalancing resumes. If a second node fails before under-replicated tablets are rebuilt, those tablets become unavailable until a replica is recreated.

Graceful decommission (rollback)

Adding a node changes cluster state. To remove the new node safely, move its replicas off first:

yb-admin -master_addresses <master-1>:7100,<master-2>:7100,<master-3>:7100 \
  change_tablet_server_state <new-node-uuid> DECOMMISSIONED

Wait for the tablet count on that node to reach zero in the UI, then stop yb-tserver. Do not kill the process while it still hosts replicas.

Limitations and version notes

• Rebalancing speed and default throttling differ across YugabyteDB 2.x minor releases. Confirm values for your installed version.
• Tables with a user-defined replication factor other than 3, or explicit placement policies, may not rebalance onto the new node unless the policy includes its zone or region.
• Leader rebalancing can lag behind replica placement by several minutes.
• This guide assumes a vanilla RF=3 universe without encryption-at-rest, read replicas, or xCluster replication.

Practical verification checklist

Check	Command / Location	Pass criteria
Node registered	Master UI or yb-admin list_tablet_servers	New node shows LIVE
Replicas arriving	Master UI tablet-servers view or get_tablet_locations	New node replica count > 0 and growing
Distribution evenness	get_tablet_locations <table-id> across several tables	Replica sets include new node UUID for a sample of tablets
Read/write availability	psql or cqlsh test during rebalancing	Zero connection errors; latency near baseline
No under-replicated tablets	Master UI under-replicated column	Zero after rebalancing settles

Run the checklist every 5 minutes until replica counts stabilize, typically 10 to 30 minutes for a few hundred tablets. If any check fails, use the Recovery options section.