Configure Per‑Guest ZFS Replication in Proxmox VE for Disaster Recovery

Desired outcome

Create a per‑guest ZFS replication job that keeps a crash‑consistent copy of a VM or container on a second cluster node, enabling fail‑over without shared storage or Ceph.

Prerequisites

• Both nodes belong to the same Proxmox VE cluster (version 5.0 or newer).
• The guest’s disks reside on a local ZFS storage on the source node.
• On the target node a ZFS storage entry with the identical storage ID exists and is not marked as shared.
• Every disk of the guest must be on a ZFS storage that is present on the target node.
• For true fail‑over the guest should not use node‑bound devices such as PCIe passthrough.
• You have root or pveadmin privileges on any cluster node.

Procedure

Via the GUI

1. In the Proxmox web UI go to Datacenter → Replication.
2. Click Add.
3. Set VM ID to the guest’s identifier (e.g. 101).
4. Choose Target node (the node that holds the replica).
5. Select the Storage that matches the source ZFS storage ID.
6. Define a Schedule using systemd calendar syntax; for a 15‑minute RPO enter */15.
7. Leave Enable job checked and click Add.

Via the CLI

Run the following command on any node (replace placeholders):

pvesr create-local-job -1  --schedule '*/15'

Example for VM ID 101 targeting node pve2:
pvesr create-local-job 101-1 pve2 --schedule '*/15'

The first execution performs a full sync; subsequent runs send only incremental ZFS send/receive streams.
Expected checks

In the GUI under Datacenter → Replication verify that the job shows Last run with a green status and note the timestamp.
Run pvesr list to see the job definition:
pvesr list

On both source and target nodes list the replication snapshots:
zfs list -t snapshot | grep __replicate_

You should see a pair of snapshots with the same name containing the VMID and job number, e.g. rpool/data/vm-101-disk-0@__replicate_101-1_2026-10-13_04-00-00 on each node.
Optionally trigger a manual run to validate the pipeline:
pvesr run 101-1

Check the task log for any errors; a successful run ends with TASK OK.

Recovery options
Planned fail‑over (migration)
If you want to move the guest to the replica node:

In the GUI select the guest and choose Migrate.
Select the target node and click Migrate.
After migration the replication job automatically reverses direction, keeping the original node as the new replica.

Emergency fail‑over after source node loss

Restore cluster quorum. In a two‑node cluster you can temporarily run on the surviving node:
pvecm expected 1

Start the guest on the target node (the replica). Because the last replicated dataset is already present, the guest boots with the data from the last successful replication run.
If HA is enabled, the CRM will perform this step automatically once quorum is regained.

Limitations and operational cautions

Replication is asynchronous; any writes committed after the last successful run are lost on fail‑over. Size the schedule to meet your required RPO.
Never delete snapshots whose names contain __replicate_ manually; doing so breaks the incremental chain and forces a full resync.
Before moving a guest’s disk to a non‑ZFS storage or destroying the guest, remove the replication job first.
Ensure the ZFS feature set on source and target pools is compatible; a much older target pool may reject the send stream.
Frequent incrementals can saturate the cluster network; consider isolating replication traffic on a dedicated VLAN or using tc to rate‑limit.
In a two‑node cluster a node loss causes loss of quorum; plan a QDevice or third node, and treat pvecm expected as an emergency measure with split‑brain risk.

After completing the steps above you have a working per‑guest ZFS replication job that can be verified via the GUI, pvesr list, and matching __replicate_ snapshots on both nodes.