Choosing a Site-to-Site VPN Protocol for pfSense: IPsec, OpenVPN, and WireGuard

The Site-to-Site Decision Problem

Selecting a VPN protocol for connecting two pfSense gateways is rarely about which one is "best," but rather which one fits your hardware constraints and network topology. The wrong choice leads to CPU saturation, unstable tunnels during rekeying, or an inability to traverse restrictive ISP firewalls.

The primary trade-off in pfSense involves the balance between throughput (kernel vs. user space), compatibility (standardization), and configuration complexity.

Protocol Comparison Matrix

Feature	IPsec (IKEv2)	OpenVPN	WireGuard
Execution Space	Kernel	User Space	Kernel
Throughput	Highest (with AES-NI)	Moderate to Low	High
Config Complexity	High (Phase 1 & 2)	Moderate	Low
NAT Traversal	Moderate (NAT-T)	Excellent (UDP/TCP)	Good (UDP)
Standardization	Industry Standard	De Facto Standard	Modern/Opinionated

Evaluating the Trade-offs

IPsec: The Performance Heavyweight

IPsec is the native choice for high-bandwidth site-to-site links. Because it operates in the kernel, it avoids the overhead of moving packets between kernel and user space. If your pfSense hardware supports AES-NI (Advanced Encryption Standard New Instructions), IPsec can often saturate a Gigabit link with minimal CPU impact.

The cost is complexity. You must match Phase 1 (Authentication/Key Exchange) and Phase 2 (Data Encryption) parameters exactly on both ends. A single mismatch in the Lifetime or Encryption Algorithm will prevent the tunnel from establishing.

OpenVPN: The Versatile Workhorse

OpenVPN is the most resilient protocol for unstable or restrictive environments. Because it can run over TCP, it can bypass firewalls that block non-standard UDP traffic. However, because it runs in user space, it is significantly more CPU-intensive than IPsec or WireGuard.

Warning: Avoid using OpenVPN over TCP on lossy WAN links. This can lead to "TCP Meltdown," where the inner TCP stream and outer TCP tunnel both attempt retransmissions, causing throughput to collapse.

WireGuard: The Modern Alternative

WireGuard provides a "middle ground" with high performance and simple configuration. It uses a fixed set of modern ciphers, removing the "negotiation phase" that makes IPsec difficult to configure. In pfSense, WireGuard is typically installed as a package.

The limitation is its lack of advanced enterprise features. It does not support the same level of dynamic routing or complex multi-site topologies as IKEv2 or OpenVPN without additional routing software (like FRR).

Implementation: Validating a WireGuard Tunnel

Given its balance of speed and simplicity, WireGuard is often the preferred choice for modern pfSense deployments. Here is how to validate a basic tunnel between two pfSense nodes.

Configuration Requirements

• Permissions: Administrator access to the pfSense WebGUI.
• Firewall Rules: Open the WireGuard UDP port (default 51820) on the WAN interface of both peers.
• Routing: Define the "Allowed IPs" to include the remote subnet (e.g., 192.168.20.0/24).

Verification Steps

Run these checks to ensure the tunnel is passing traffic and not just "up":

1. Handshake Check: Navigate to VPN > WireGuard > Status. Look for a "Latest Handshake" timestamp within the last few minutes. If no handshake is present, check WAN firewall rules.
2. Path Validation: From a client on the local subnet, run a traceroute to a host on the remote subnet. The first hop should be the local pfSense gateway, and the second should be the remote pfSense gateway.
3. Throughput Test: Run iperf3 between two hosts across the tunnel.

   # On remote host (Server)
   iperf3 -s
   
   # On local host (Client)
   iperf3 -c [remote_ip]

Operational Limitations

Regardless of the protocol, remember that VPNs introduce MTU (Maximum Transmission Unit) overhead. If you experience "stalling" on large file transfers or web pages that won't load, you likely have an MTU mismatch. A practical fix is to implement MSS Clamping under VPN > IPsec/OpenVPN/WireGuard settings to ensure packets fit within the tunnel encapsulation.

Rollback Procedure

If a new VPN configuration disrupts existing WAN traffic or creates a routing loop:

1. Disable the VPN interface or service immediately via the WebGUI.
2. Remove any static routes created specifically for the VPN tunnel under System > Routing.
3. Delete the associated firewall rules on the WAN and VPN interfaces to return to the previous security posture.