Implementing High‑Availability with CARP on pfSense: Minimal Design, Trust Boundaries, and Operational Checks

Problem & Takeaway

Organizations that need uninterrupted firewall service often turn to pfSense’s CARP (Common Address Redundancy Protocol). The challenge is to build a small, reliable HA pair that keeps the virtual IP (VIP) reachable without introducing split‑brain or configuration drift. The key is a minimal hardware footprint, a shared state store, and health‑check‑driven priority adjustments. The design below shows how to achieve that with two identical pfSense nodes.

Requirements

• Two pfSense installations (same firmware release, e.g. 2.6.0‑RELEASE) on separate physical hosts.
• Each host must have:
  • One LAN interface (e.g., em0) connected to the internal network.
  • One WAN interface (e.g., em1) connected to the Internet.
  • A dedicated CARP interface (e.g., em2) that shares the same subnet as the LAN.
• Shared storage accessible to both nodes (NFS or SMB), mounted at /var/db/pfSense/rsync.
• SSH access for rsync and command‑line checks.

Smallest Suitable Design

The minimal HA pair consists of the two nodes described above. The CARP VIP (e.g., 192.168.1.10) is configured on the LAN subnet. Both nodes run the same config.xml pulled from the shared store. A simple health‑check script on the active node monitors the WAN link and, if it goes down, temporarily lowers its CARP priority so the standby automatically takes over.

CARP Configuration (example)

# On both nodes, edit /etc/rc.conf and add:
# CARP interface
ifconfig_carp0="inet 192.168.1.10 netmask 255.255.255.0"
# Virtual MAC (must match on both nodes)
ifconfig_carp0="vlan 0"
# CARP password (shared secret, 8–32 chars)
carp_password="MySecretPwd"
# Priority: active higher than standby
carp_priority=200   # on active
carp_priority=100   # on standby

Replace the priority values accordingly. The default CARP heartbeat interval is 5 s; you can adjust it with carp_interval if needed.

Shared Config Sync (rsync over SSH)

# On each node, add to /etc/rc.conf:
rsyncd_enable="YES"
rsyncd_listenaddr="0.0.0.0"
rsyncd_port="873"
rsyncd_banlist="/etc/rsyncd.banlist"
rsyncd_maxconnections="2"
rsyncd_readonly="NO"
rsyncd_usechroot="NO"
rsyncd_authusers="pfsense"
rsyncd_secretsfile="/etc/rsyncd.secrets"
# In /etc/rsyncd.secrets:
# pfsense:your_password

Set up a cron job that runs rsync -avz /etc/ssl /etc/rc.d /etc/config.xml /var/db/pfSense/rsync every minute. This keeps both nodes in sync.

Trust & Data Boundaries

• CARP VIP is the only public address exposed to clients; the real node IPs remain hidden.
• Shared storage must be isolated from the WAN to avoid leaking configuration changes to attackers.
• SSH keys for rsync are stored on both nodes with strict permissions (600).
• CARP passwords are shared but kept on each node’s /etc/rc.conf only; they should not be in version control.

Operational Sanity Checks

1. Dashboard: In the pfSense WebUI, go to Status > CARP. The active node should show the VIP as Active, the standby as Standby.

2. Ping test: From a client on the LAN, ping the VIP. While the active node is up, the ping should succeed; after a controlled shutdown of the active WAN interface, the ping should continue without loss.

3. pfctl trace: SSH into the active node and run pfctl -vv -t carp. You should see CARP packets being sent every 5 s. After disabling the WAN, the output should show the standby taking over.

4. Config parity: Verify /var/db/pfSense/rsync/config.xml is identical on both nodes using diff /var/db/pfSense/rsync/config.xml /var/db/pfSense/rsync/config.xml (replace paths accordingly).

Failure Modes & Mitigation

• Network partition between nodes: If the LAN link to one node fails, the other will still advertise the VIP. Mitigation: use a separate VLAN for CARP traffic or a dedicated link.
• Mis‑configured CARP password: The standby will never join the group. Mitigation: verify the password on both nodes before enabling HA.
• Stale shared config: If the shared storage is unreachable during failover, the standby will use its last cached config. Mitigation: use an NFS mount that stays online even if one node’s network interface fails.
• Split‑brain: If both nodes think they are active (e.g., due to priority mis‑match), the VIP may be duplicated. Mitigation: keep priority values strictly higher on the active node and use a health‑check script to lower it when the WAN is down.
• Firmware mismatch: Different pfSense releases can cause CARP incompatibility. Mitigation: keep both nodes on the same firmware version.

Design Pivot Points

• WAN failure: The current design lowers the active node’s priority, letting the standby take over. If you need zero‑downtime for critical services, consider adding a hot‑standby script that fails over faster than the default 5 s interval.
• Shared storage outage: If the NFS share is lost, the standby will use its local copy. For higher resilience, use a redundant storage solution (e.g., DRBD) or keep a copy on a USB stick that both nodes can mount.
• Single‑link risk: Using the same physical link for WAN and CARP traffic creates a single point of failure. Pivot to a dedicated CARP interface on a separate NIC if available.
• Scale beyond two nodes: For a cluster of three or more, you can extend the CARP VIP range and use a load‑balancing algorithm in pf (e.g., table and ipnat rules). This requires additional configuration and careful priority management.

Practical Verification Checklist

Check	Command / UI	Expected Result
VIP status	Status > CARP (WebUI)	Active on one, Standby on the other
Ping continuity	Ping 192.168.1.10 from client	No packet loss across failover
CARP traffic	SSH: pfctl -vv -t carp	Packets every 5 s; priority change on WAN loss
Config parity	SSH: diff /var/db/pfSense/rsync/config.xml /var/db/pfSense/rsync/config.xml	No differences
Shared storage health	SSH: df -h /var/db/pfSense/rsync	Mounted and writable on both nodes

Conclusion

By keeping the HA design minimal—two identical pfSense nodes, a shared NFS store, a dedicated CARP interface, and a health‑check‑driven priority script—you can achieve reliable failover with predictable behavior. Regularly verify the configuration, monitor CARP status, and keep both nodes on the same firmware. When conditions change (e.g., adding a third node or switching to a different shared storage), revisit the trust boundaries and operational checks to maintain the system’s integrity.