Preventing State Loss in pfSense CARP High-Availability Clusters

The Failover Gap: When Redundancy Isn't Seamless

Deploying a second pfSense firewall for high availability (HA) often leads to a false sense of security. While Common Address Redundancy Protocol (CARP) allows a backup node to take over a Virtual IP (VIP) almost instantly, it does not automatically ensure that active connections survive the transition. Without precise state synchronization, a failover triggers a mass disconnection of all active TCP sessions, forcing every client to re-authenticate or restart downloads.

The goal of a robust CARP setup is to achieve stateful failover, where the backup node already knows about every active connection on the primary node before the failure occurs.

How CARP Handles Identity and Traffic

CARP operates at the data link layer. It uses a Virtual IP (VIP) that is shared between two or more pfSense units. These units send multicast advertisements to one another; as long as the primary node is "heartbeating," the backup node remains passive. If the heartbeat stops, the backup node assumes the VIP and begins routing traffic.

To avoid network instability, you must assign a unique Virtual Host ID (VHID) to each redundancy group. A VHID collision—where two different CARP groups use the same ID on the same subnet—can cause ARP storms, leading to intermittent connectivity or complete network collapse.

Synchronizing the State Table

A VIP transition handles the address, but the pf (packet filter) kernel module handles the session. If the backup node has an empty state table, it will see incoming packets for existing connections as "out-of-state" and drop them based on default security rules.

To prevent this, you must configure pfsync. This service replicates the state table from the primary to the secondary node over a dedicated synchronization interface. For this to work effectively, both nodes must have identical firewall rule hierarchies. If Node A allows port 443 but Node B does not, the state synchronization will occur, but the backup node will still drop the traffic upon failover.

Worked Example: Configuring a Sync Interface

For a stable deployment, do not run synchronization traffic over your WAN or LAN interfaces. Use a dedicated physical link or a tagged VLAN specifically for HA traffic.

Configuration Steps:

1. Assign Sync Interface: On both nodes, go to Interfaces > Assignments and designate a dedicated port as the "SYNC" interface. Enable the interface and assign a static IP (e.g., 10.0.0.1/30 on Primary, 10.0.0.2/30 on Secondary).
2. Configure XMLRPC Sync: On the Primary node, navigate to System > High Avail. Sync. Enter the IP of the Secondary node and a synchronization password. This ensures that firewall rules and DHCP leases are mirrored automatically.
3. Define the CARP VIP: Navigate to Firewall > Virtual IPs. Add a new IP, select CARP, and assign a unique VHID (e.g., 10).

Verification Command: To check the current status of the CARP nodes from the shell (SSH), run:

# Run on either node to see master/backup status
carpctl show

Expected Result: One node should report MASTER and the other BACKUP for the specific VHID. If both report MASTER, you have a "split-brain" scenario, usually caused by a failure in the heartbeat link.

Trade-offs and Limitations

State synchronization is not free. Under extremely high connection loads (thousands of concurrent states per second), the synchronization traffic can consume significant CPU and bandwidth on the sync interface. In these environments, you may experience "convergence stall," where the backup node cannot keep up with the primary's state changes.

Furthermore, CARP does not synchronize everything. While firewall rules and states are mirrored, certain package-specific data (like some Suricata or Snort signatures/logs) may not sync perfectly, requiring manual verification after a failover event.

Validating the Failover

The only way to guarantee your HA setup works is to force a failure. Perform a planned disruption by physically disconnecting the WAN cable from the Primary node. Monitor a continuous ping to a public DNS server from a client machine. You should see a brief spike in latency (usually 1-3 missed pings) before the Secondary node takes over and traffic resumes without the client needing to reconnect.

Rollback: To return to normal operation, reconnect the Primary WAN. The Primary node will typically resume the MASTER role based on its configured priority (advertisement interval).