Phoenix backup via HBase snapshots: capturing secondary indexes and SYSTEM.CATALOG in one consistent restore

Answer to the Core Question

There is no native, atomic HBase snapshot that spans multiple tables unless you use HBase table‑group snapshots (available in HBase 2.5+). Without table‑group support you must pause writes (or stop DDL) while you snapshot the Phoenix data table, each of its index tables, and SYSTEM.CATALOG with the same snapshot name.

Likely explanation vs. confirmed facts

Likely explanation: If your cluster runs HBase 2.5 or newer and you have defined a table group that contains the data table, all its index tables, and SYSTEM.CATALOG, a single group snapshot will capture a consistent view without stopping traffic.

Confirmed facts: HBase snapshots are table‑level objects. Phoenix stores each secondary index as a separate HBase table (e.g., MYTABLE_IDX_1) and keeps schema metadata in SYSTEM.CATALOG. Taking independent snapshots at different times can leave the data table and its indexes out of sync if writes are in flight.

Steps for a consistent backup (write‑pause path)

1. Identify the tables to snapshot:

   hbase shell
   > describe 'MYTABLE'
   > SELECT INDEX_NAME FROM SYSTEM.INDEX WHERE TABLE_NAME='MYTABLE';

2. Quiesce writes to the Phoenix table and its indexes (e.g., stop the application or issue disable in HBase shell) and flush memstores:

   hbase shell
   > disable 'MYTABLE'
   > flush 'MYTABLE'
   > # repeat disable/flush for each index table discovered above
   > disable 'SYSTEM.CATALOG'
   > flush 'SYSTEM.CATALOG'

3. Snapshot each table using a common snapshot name (e.g., phoenix_backup_2026_10_14):

   hbase shell
   > snapshot 'MYTABLE', 'phoenix_backup_2026_10_14'
   > snapshot 'MYTABLE_IDX_1', 'phoenix_backup_2026_10_14'
   > snapshot 'MYTABLE_IDX_2', 'phoenix_backup_2026_10_14'
   > snapshot 'SYSTEM.CATALOG', 'phoenix_backup_2026_10_14'

4. Re‑enable writes:

   hbase shell
   > enable 'MYTABLE'
   > # repeat enable for each index table and SYSTEM.CATALOG

5. Verify that all snapshots exist and share the same name/timestamp:

   hbase shell
   > snapshot -list

Verification after restore

• Check that every index for the table is marked ACTIVE in SYSTEM.INDEX:

  SELECT * FROM SYSTEM.INDEX WHERE TABLE_NAME='MYTABLE';

• Optionally compare row counts of the data table and an index table on the indexed column(s); they should be equal for a unique index.
• Run EXPLAIN on a selective query that should use the index and confirm the plan shows an INDEX SCAN (or RANGE SCAN) for the expected index name:

  EXPLAIN SELECT * FROM MYTABLE WHERE COLUMN1 = 'value';

Cross‑cluster recovery with ExportSnapshot

For disaster recovery or migration, export each snapshot to a shared location and import them on the target cluster.

1. Export the snapshots:

   hbase shell
   > export_snapshot 'phoenix_backup_2026_10_14', '/mnt/backup/phoenix_backup_2026_10_14'

2. On the target cluster, ensure the same namespace exists (HBase does not snapshot namespace metadata), then restore:

   hbase shell
   > restore_snapshot 'phoenix_backup_2026_10_14'

3. If the target Phoenix version differs, run ALTER TABLE … REBUILD INDEX or update catalog compatibility after restore.

Diagnostic question

Do you have HBase 2.5 or newer and have you defined a table group that includes the data table, all its index tables, and SYSTEM.CATALOG? If yes, you can skip the write‑pause step and take a single group snapshot; otherwise, follow the pause‑and‑snapshot procedure above.