Worker threads vs shared SSL_CTX: handshake latency under concurrent TLS load

Answer to the Core Question

For concurrent TLS handshakes, a single SSL_CTX with per-thread SSL objects is functionally sufficient and the standard design. However, under high concurrent load, the internal session cache, ticket-key callbacks, and OCSP stapling mechanisms are protected by a single mutex. This lock becomes a measurable bottleneck when handshake rates reach high thresholds (typically >10k/s) on multi-core systems. Per-worker SSL_CTX instances eliminate this lock contention but disable cross-worker session resumption and increase memory overhead due to duplicated certificate chains and caches.

Confirmed Facts (OpenSSL 1.1.1/3.x)

• Session Cache: Internal session cache operations (lookup, insert, evict) are serialized by a mutex.
• Callbacks: Ticket-key and OCSP stapling callbacks are executed while holding the SSL_CTX lock.
• OpenSSL 3.x Providers: Algorithm fetching uses a global provider store. The first fetch requires a write lock; subsequent fetches use fast read-only reference increments.
• Randomness: RAND_bytes calls are lock-free after initial seeding because OpenSSL 1.1.1+ uses per-thread DRBG instances.
• Resource Cost: Per-worker SSL_CTX instances duplicate the X509_STORE, certificate chains, and DH parameters, typically costing 200–500 KiB per instance plus cache memory.

Likely Explanation for Latency Differences

Latency spikes under concurrent load are usually caused by mutex contention rather than algorithm computation. When multiple threads attempt to validate or store sessions simultaneously, they queue for the SSL_CTX lock. This effect is severely amplified if ticket-key or OCSP callbacks perform blocking I/O (such as calling an HSM or network service), as the lock is held for the duration of the call, effectively serializing all handshakes across the process.

In contrast, OpenSSL 3.x provider fetches are a "warm-up" cost. Once the required EVP algorithms (e.g., AES-GCM) are fetched and reference-counted, they do not contribute to per-handshake latency.

When to Prefer a Shared SSL_CTX

• Handshake rates are below the contention threshold (generally < 10k/s).
• Cross-worker session resumption is required for client performance.
• Callbacks are lightweight, in-memory operations.

When Per-Worker SSL_CTX is Necessary

• Extreme handshake concurrency on high-core-count CPUs.
• Callbacks involve external I/O or high-latency HSM calls.
• Session resumption is handled by an external store (e.g., Redis), rendering the internal SSL_CTX cache redundant.

Practical Steps to Decide

1. Benchmark Latency: Use openssl s_server -www -multi N and openssl s_time -connect ... -new with varying thread counts (1, 2, 4, 8) to identify the saturation point.
2. Analyze Contention: Use perf lock record to determine if CRYPTO_THREAD_lock or similar mutexes are the primary source of CPU wait time.
3. Verify Version: Run openssl version -v to ensure you are using 1.1.1k+ or 3.0.8+ to benefit from known lock optimizations.
4. Trace Providers: In OpenSSL 3.x, use export OSSL_TRACE=PROVIDER to confirm that algorithm fetches occur only during initialization.

Missing Diagnostic Detail

To provide a definitive recommendation, what is the expected peak number of concurrent TLS handshakes per second the application must support?