SSL_CTX_set_info_callback notification noise during TLS handshakes
0 reputation · 10 Mar 2024, 18:49 UTC
0 reputation · 10 Mar 2024, 18:49 UTC
Implementing a monitoring system to capture TLS handshake failures requires balancing detailed visibility with the volume of informational events. Using SSL_CTX_set_info_callback allows an application to track the progress of a connection, but the callback triggers for both successful state transitions and error conditions.
Because OpenSSL utilizes a thread-local error queue, failures captured via ERR_get_error may persist across multiple checks if the queue is not explicitly drained. This creates a risk of duplicate alerts or stale error reporting when combined with the high-frequency events of the info callback.
There is uncertainty regarding the most efficient way to filter out routine handshake progress to isolate only actionable certificate verification failures without missing transient TLS alerts.
where and ret parameters in the info callback be used to distinguish critical failures from informational noise?35275 reputation · 11 Mar 2024, 06:01 UTC
To isolate critical failures from informational noise in the SSL_CTX_set_info_callback, filter the ret parameter first and then narrow the scope using the where parameter. In OpenSSL 3.x, a ret value of 1 typically indicates a successful state transition, while a value of 0 or less indicates an error or a need for more data.
The where parameter identifies the specific stage of the TLS state machine. To reduce noise, ignore all callbacks where ret == 1. To isolate actionable failures, monitor for ret <= 0 specifically during these states:
SSL_CB_HANDSHAKE_DONE: Confirms the final result of the handshake.SSL_CB_HANDSHAKE_START: Useful for timing, but usually noise unless it fails immediately.Note that certificate verification failures often trigger a failure in the general handshake flow rather than a specific where constant dedicated solely to X509 validation. Therefore, the most reliable pattern is to trigger your error-logging logic whenever ret <= 0 is encountered during the handshake phase.
Because OpenSSL maintains a thread-local error queue, failing to drain it will cause stale errors to be reported in subsequent callbacks. The recommended pattern is to loop through ERR_get_error() until the queue is empty.
void info_callback(const SSL *s, void *arg, int where, int ret) {
if (ret <= 0) {
unsigned long err;
// Drain the queue to prevent duplicate/stale alerts
while ((err = ERR_get_error()) != 0) {
char buf[256];
ERR_error_string_n(err, buf, sizeof(buf));
// Log the specific error (e.g., certificate expired)
fprintf(stderr, "TLS Error: %s\n", buf);
}
}
}
This approach assumes you are using OpenSSL 3.x and that the callback is executing within the same thread that performed the SSL operation. If your architecture offloads SSL processing to a separate thread pool, ERR_get_error() must be called in the context of the thread that encountered the error.
Diagnostic Detail Needed: Are you using a custom SSL_CTX_set_verify callback in addition to the info callback? If so, certificate-specific failures may be intercepted there before they reach the info callback's ret check.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 11 Mar 2024, 05:33 UTC
To further refine filtering, it is important to note that the where parameter is a bitmask. While checking for ret <= 0 identifies an error state, you can use bitwise AND operations to isolate specific phases of the handshake that matter to your monitoring.
For example, if you are only concerned with certificate-related issues, filter for SSL_CB_CERT_VERIFY. This allows your application to ignore failures occurring during protocol negotiation or cipher suite exchange, which might be handled by lower-level retry logic rather than your X509-specific monitor.
Additionally, when draining the error queue, ensure you only call ERR_get_error() when SSL_CB_RETURN_ERROR is present. Draining it during routine informational transitions might inadvertently clear errors that were intended to be caught later in the connection lifecycle.