pfSense split-DNS overrides and TLS certificate validation limits

Isolating DNS and TLS Failures

To determine the origin of a connection failure when using pfSense host overrides, you must isolate the resolution phase from the handshake phase. Because these processes occur sequentially, a failure at the DNS layer prevents the TLS handshake from ever initiating.

Step 1: Verify DNS Resolution and DNSSEC

First, confirm that the client is receiving the overridden IP and not a SERVFAIL. A SERVFAIL in the pfSense DNS Resolver often indicates a DNSSEC validation failure for the zone.

• Test: Run dig @pfSense_IP hostname.com or nslookup hostname.com from a client.
• Analysis: If the result is SERVFAIL, the issue is DNSSEC. If it returns the public IP instead of your override, the Resolver is ignoring the override or forwarding the query. If it returns the private IP, the DNS layer is functional.

Step 2: Inspect the TLS Handshake

If DNS returns the correct private IP, use openssl to bypass the browser's generic error messages and inspect the presented certificate.

openssl s_client -connect [Private_IP]:443 -servername [hostname.com]

Analyze the output for two specific failure modes:

1. Name Mismatch: If the certificate is valid but the Subject Alternative Name (SAN) does not include hostname.com, the client will reject the connection.
2. Trust Chain Failure: If the certificate is issued by a private CA and the client lacks the root certificate, you will see an unknown ca or self signed certificate error.

Strategic Recommendation: SAN Alignment vs. Trust Stores

Prioritize matching certificate SAN entries to overridden hostnames.

Aligning the certificate to the DNS identity is the most reliable and maintainable approach. Adjusting per-service trust stores or deploying root CAs to every client device creates significant administrative overhead and scales poorly. By ensuring the internal server presents a certificate with a SAN that matches the split-DNS override, you maintain a consistent identity across the network boundary.

Summary of Logic

Symptom	Likely Cause	Verification Command
No IP returned / SERVFAIL	DNSSEC Failure	dig @pfSense_IP [host]
Wrong IP returned	Override Misconfiguration	nslookup [host]
SSL_ERROR_BAD_CERT_DOMAIN	SAN Mismatch	openssl s_client
SEC_ERROR_UNKNOWN_ISSUER	Missing Root CA	openssl s_client

Diagnostic Detail Needed: Are the clients experiencing these failures using a browser with HSTS enabled for the public domain? HSTS can force strict validation that ignores certain local trust overrides.