Hostname verification transition in OpenSSL 1.1.0

Question

OpenSSL 1.1.0 introduced the X509_VERIFY_PARAM_set1_host and SSL_set1_host APIs to implement RFC 2818 hostname verification. Unlike previous versions where some tools performed implicit checks, this verification is not enabled by default in the library; applications must explicitly set the target host name to activate the check.

This change affects how the s_client tool operates, as it now requires the -servername option to perform hostname verification. For developers migrating from OpenSSL 1.0.2, this shift from implicit to explicit configuration creates a boundary where connections may succeed during the handshake but fail to validate the identity of the peer if the API is not called correctly.

What is the precise behavior when SSL_set1_host is called with a trailing dot in the DNS name? Does the library treat the absence of an explicit host name as a bypass of all hostname checks, or is there a fallback mechanism?

Siddharth · Answer

Hostname Verification Logic in OpenSSL 1.1.0
In OpenSSL 1.1.0, the library treats the absence of an explicit host name as a complete bypass of hostname verification. If neither SSL_set1_host nor X509_VERIFY_PARAM_set1_host is called, the library validates the certificate chain and expiration but does not verify that the certificate belongs to the peer you are connecting to. This means any valid certificate issued by a trusted CA will be accepted, regardless of the domain name it covers.

Trailing Dots in DNS Names
When SSL_set1_host is called with a trailing dot (e.g., "example.com."), the library treats it as a literal string for comparison against the Subject Alternative Name (SAN) or Common Name (CN). Because the trailing dot signifies a Fully Qualified Domain Name (FQDN) in DNS standards, the library handles it as part of the identity match logic. If the certificate's SAN contains the name without the trailing dot, the verification may fail depending on the specific matching rules applied by the version's internal string comparison.

Migration and Verification Steps
To ensure identity verification is active during the transition from 1.0.2 to 1.1.0, follow these implementation steps:

Explicit Configuration: Call SSL_set1_host(ssl, "hostname") before initiating the handshake.
Verification: Use the s_client tool to test the behavior. A connection without -servername may succeed in the handshake but will fail to perform the identity check, whereas -servername triggers the necessary internal flags for verification.

Assumptions and Limitations
This behavior assumes a standard build of OpenSSL 1.1.0. We assume the application is relying on the library's internal verification rather than a custom X509_STORE_CTX callback. Because trailing dot handling can occasionally interact with system-level resolver logic in certain OS environments, it is recommended to test the specific certificate format (SAN vs. CN) used by your peer.

Diagnostic Detail Needed: Are you using a custom verification callback (SSL_set_verify) in addition to SSL_set1_host? If so, the callback may override or conflict with the library's automated hostname check.

Hostname verification transition in OpenSSL 1.1.0

1 answer

Hostname Verification Logic in OpenSSL 1.1.0

Trailing Dots in DNS Names

Migration and Verification Steps

Assumptions and Limitations

Your perspective belongs here.

0 question comments