Vitess VReplication: Built-in CDC and Zero-Downtime Migrations

The problem: you need change data capture, materialized views, or schema migrations without adding Debezium, Kafka Connect, or a separate migration tool

Most MySQL shops eventually hit the same wall: they need to stream row changes to an analytics store, keep a denormalized reporting table in sync, or run ALTER TABLE on a 2 TB table without locking the primary for hours. The usual answer is to bolt on a CDC pipeline — Debezium, Maxwell, or a custom binlog reader — plus a transformation layer for filtering or masking. That adds operational surface: more components to deploy, monitor, and secure.

Vitess ships a different answer. VReplication is a pull‑based streaming replication engine built into the control plane. It reads binlogs directly from source tablets, applies optional row filtering and column projection, and writes to target tables – all driven by declarative VSchema rules and managed through vtctlclient. No external CDC connector required.

How VReplication works under the hood

Each VReplication stream runs on a target tablet. The streamer process connects to a source tablet, requests binlog events starting from a stored GTID position, filters and projects rows according to the workflow’s SELECT statement, and applies them to the target table. The GTID position is persisted in the _vt.vreplication system table, so streams are independently resumable after restarts or failover.

Because streams are tablet‑scoped, not cluster‑scoped, adding a new replica only means starting a new stream on that tablet – no central coordinator becomes a bottleneck. A single workflow can fan‑out from one source to many targets (CDC to Kafka, a reporting replica, a search index) or fan‑in from many shards into one analytics table.

Declarative workflow definition via VSchema

You define what to replicate in the VSchema, not in imperative code. A workflow spec looks like:

{
  "vreplication_workflows": [
    {
      "workflow": "user_cdc",
      "source": "keyspace:user",
      "target": "keyspace:user_analytics",
      "blp_checkpoint": "",
      "cells": ["zone1"],
      "tablet_types": ["REPLICA"],
      "workflow_type": "MaterializedView",
      "streaming": true,
      "query": "SELECT user_id, email_hash, created_at FROM user WHERE is_active = 1"
    }
  ]
}

The planner rewrites the query so it automatically fans out to the correct source shards based on the sharding key (vindex). You don’t write per‑shard logic.

Worked example: zero‑downtime schema migration on a 500 GB table

1. Create the workflow: vtctlclient StartVReplication --workflow=orders_add_idx --source=commerce:orders --target=commerce:orders_shadow --query="SELECT * FROM orders" --workflow_type=OnlineDDL
2. The controller creates a shadow table orders_shadow with the new index definition.
3. Copy phase streams existing rows. Throttle with --vreplication-copy-phase-rows-per-second=50000 to avoid saturating source IO. Monitor VReplicationCopyPhaseRowsCopied metric.
4. Catch‑up phase streams binlog events while the ALTER runs on the shadow table.
5. Dry‑run validation: vtctlclient VReplicationExec orders_add_idx copy-traffic validates row counts and checksum matches before the real cut‑over.
6. Atomic swap: the controller renames orders_shadow to orders and drops the old table – no metadata lock storm.

Run vtctlclient VReplicationExec orders_add_idx SHOW CREATE TABLE orders on the target to verify the new index exists.

Trade‑offs and limitations you’ll hit in production

• Cross‑cell replication is not native. Streaming across regions requires a VPN or peered network; binlog lag grows with RTT. Plan latency budgets accordingly.
• DDL on source tables must stay backward‑compatible. Adding a column works; dropping a column breaks the stream until you update the workflow’s SELECT list and restart with vtctlclient UpdateVReplicationWorkflow.
• No transactional consistency across shards. Each shard streams independently, so a distributed transaction may appear partially applied in the target until all shards catch up.
• Metadata contention at scale. The _vt.vreplication table becomes a hotspot with hundreds of concurrent workflows. Batch workflow creation.
• Hard requirements. ROW‑format binlogs and GTID_MODE=ON. Statement‑based replication or anonymous GTIDs cause the streamer to error out at startup.

Actionable next step: validate in a local cluster

Spin up the Vitess local example (vitess/examples/local), create a workflow with vtctlclient StartVReplication, insert rows on the source shard, then query the target table and _vt.vreplication.time_updated to confirm sub‑second lag. Enable Prometheus metrics on vttablet (/metrics) and graph vreplication_lag_seconds and vreplication_streamer_errors_total during a test resharding cut‑over. The Vitess docs section “VReplication User Guide” (versioned per release) has the exact CLI flags and VSchema syntax used above.