OpenSSL 3.x Provider and TLS 1.3 Configuration for FIPS or Compatibility

The decision you are actually making

You need TLS termination on OpenSSL 3.x with two mutually exclusive goals: maximum compatibility and latency with modern algorithms, or auditable FIPS‑approved cryptography. The useful takeaway is to run TLS 1.3 only with a restricted group list on the default provider for production speed, and switch to the FIPS provider with a reduced cipher and group set when regulatory approval is required. Never enable the legacy provider for new traffic.

Constraints that drive the choice

• Regulatory approval: FIPS 140‑3 requires the FIPS provider initialized and active, with no non‑approved algorithms reachable.
• Client compatibility: TLS 1.3 only drops older clients. TLS 1.2 with CBC and RSA PKCS#1 v1.5 must be disabled to avoid weak cipher selection.
• Latency budget: X25519 and AES‑GCM with hardware acceleration lower handshake and record overhead.
• Operational simplicity: fewer providers and explicit protocol minima reduce misconfiguration risk.

Options compared

Trade-offs

Default provider maximizes compatibility and speed and permits X25519 and secp256r1 for TLS 1.3. It does not meet FIPS 140‑3 out of the box.

FIPS provider provides an auditable algorithm set required in regulated environments. It restricts curves and removes ChaCha20‑Poly1305, increases configuration complexity, and can reduce reach for older TLS 1.2 clients if you also enforce TLS 1.3 only.

Disabling the legacy provider hides APIs but does not remove code. Applications that explicitly request weak algorithms can still access them if linked against legacy symbols.

Concrete implementation

Broad compatibility with TLS 1.3 only (default provider)

Edit the OpenSSL configuration file (typically /etc/ssl/openssl.cnf) as root or with sudo:

# openssl.cnf snippet
openssl_conf = openssl_init

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
# fips = fips_sect   # uncomment only when FIPS is required

[default_sect]
activate = 1

In the application that creates the TLS context (no special permissions beyond normal process execution):

SSL_CTX *ctx = SSL_CTX_new(TLS_server_method());
SSL_CTX_set_min_proto_version(ctx, TLS1_3_VERSION);
SSL_CTX_set_max_proto_version(ctx, TLS1_3_VERSION);
SSL_CTX_set_ciphersuites(ctx, \"TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256\");
SSL_CTX_set1_groups_list(ctx, \"X25519:secp256r1\");

When FIPS compliance is required

Enable the FIPS provider and restrict the cipher list to FIPS‑approved suites (ChaCha20‑Poly1305 is omitted). Example group list often reduces to secp256r1 only.

OSSL_PROVIDER_load(NULL, \"fips\");
OSSL_PROVIDER_load(NULL, \"default\");

Then set the same protocol minima and a FIPS‑approved cipher suite, for example:

SSL_CTX_set_ciphersuites(ctx, \"TLS_AES_256_GCM_SHA384\");
SSL_CTX_set1_groups_list(ctx, \"secp256r1\");

Risk: if the FIPS provider fails to load, OpenSSL silently falls back to the default provider, breaking compliance claims. Always verify provider activation before accepting connections.

Validation

Run on the server host to confirm which providers are loaded:

openssl list -providers

Check that the expected providers (default, and fips if enabled) show as active.

Test handshake parameters with the command‑line tools. Start a test server (you need permission to bind to the chosen port, e.g. 8443):

openssl s_server -provider default -groups X25519:secp256r1 -ciphersuites TLS_AES_256_GCM_SHA384 -tls1_3 -accept 8443

In another terminal connect with:

openssl s_client -connect localhost:8443 -tls1_3

Inspect the output for the negotiated protocol, cipher suite and key exchange group; they must match the configured minima.

In application code after the handshake:

• SSL_get_version should return TLS 1.3.
• SSL_get_current_cipher should match one of the allowed suites.
• SSL_get_peer_tmp_key (or SSL_get_curves) can be used to confirm the selected group.

Before accepting connections, verify that SSL_CTX_get0_param reflects the intended cipher suite and group list.

Limitations

The FIPS module must be correctly installed and initialized; otherwise the default provider is used. ChaCha20‑Poly1305 is not FIPS‑approved in most builds. X25519 availability under FIPS depends on the OpenSSL version and the FIPS module build. The legacy provider must remain disabled for new deployments to prevent accidental weak cipher selection.