Diagnosing OpenSSL TLS Handshake Failures: Symptoms, Checks, and Fixes

Recognizable Condition

When an OpenSSL‑based client or server aborts a TLS handshake, the tool typically prints an alert such as “handshake failure”, “certificate verify failed”, or “no shared cipher”. Recognizing the exact alert helps narrow the underlying cause.

Cause & Diagnostic Table

Observed alert	Most common cause
handshake failure	Protocol version or cipher suite mismatch
certificate verify failed	Untrusted CA, expired certificate, or hostname mismatch
no shared cipher	No overlapping cipher suite enabled on both sides
unknown ca	Missing or incorrect trust store
bad certificate (or similar)	Server sent a certificate that does not match the expected SNI

Ordered Checks

1. Confirm the OpenSSL version in use: openssl version. Note that the 1.1.1 LTS series enables TLS 1.0‑1.2 by default, while the 3.x series may have stricter defaults.
2. Test connectivity with a specific TLS version to isolate version issues: openssl s_client -connect host:port -tls1_2 (replace host and port with the target). Observe whether the handshake proceeds past the ServerHello.
3. If the version test fails, repeat with TLS 1.3 (-tls1_3) or with an explicit protocol list (-tls1, -tls1_1) to see which versions are accepted.
4. List the cipher suites offered by the client: openssl ciphers -v 'HIGH'. Then query the server’s advertised suites with openssl s_client -connect host:port -cipher. Compare the two lists for overlap.
5. Examine the server’s certificate: openssl s_client -connect host:port -servername host -showcerts. Capture the PEM block and run openssl x509 -in cert.pem -text -noout to check validity dates, issuer, and Subject Alternative Name (SAN) fields.
6. Verify SNI handling: run the same s_client command with and without -servername. A missing or incorrect SNI often triggers a “handshake failure” or “certificate verify failed” alert when the server hosts multiple virtual hosts.
7. If an hardware engine is configured, list and test it: openssl engine -t. Ensure the engine is loaded and functional before retrying the handshake.
8. Enable verbose tracing to pinpoint where the negotiation stops: openssl s_client -connect host:port -msg -trace. Look for the last printed message before the alert.

Fixes Tied to Findings

Diagnosed cause	Remedial action
Protocol version mismatch	Upgrade or downgrade the client/server to a mutually supported TLS version. Avoid enabling SSLv3 or TLS 1.0 unless absolutely necessary and only in a controlled test environment.
Cipher suite incompatibility	Adjust the cipher list on either side (e.g., openssl ciphers -v 'HIGH:!aNULL') or update the OpenSSL configuration file (openssl.cnf) to include overlapping suites. Restart the dependent service after changing the configuration.
Untrusted or expired certificate	Obtain a certificate from a trusted CA, renew the expired certificate, or add the missing CA to the client’s trust store (ca-certificates directory or a custom CAfile).
Hostname/SAN mismatch	Reissue the certificate with the correct DNS names in the SAN extension, or adjust the client’s -servername argument to match the name presented by the server.
Missing trust store	Specify the path to a valid CA bundle with -CAfile /etc/ssl/certs/ca-certificates.crt (Linux) or the appropriate bundle for your platform.
Engine misconfiguration	Verify the engine loads correctly (openssl engine -t -t engine_name). If the engine fails, disable it temporarily or correct the engine configuration file, then restart the application using OpenSSL.

Escalation Criteria

• If after verifying versions, ciphers, certificates, SNI, and engine status the handshake still fails, capture a full trace with -msg -trace and share it with the vendor or OpenSSL mailing list.
• When the failure occurs only with a specific hardware accelerator, consider testing with the engine disabled (OPENSSL_CONF pointing to a config without the engine) to isolate the issue.
• Never disable certificate verification (-verify 0) in production; reserve that option for isolated test environments only.

After applying a fix, repeat the relevant verification step (e.g., re‑run the version‑specific s_client command) and confirm that the output ends with a line similar to Verify return code: 0 (ok) and that the session is established without an alert.