Which strategy minimizes log noise for non-fatal TLS alerts in OpenSSL?

OpenSSL does not provide a native configuration flag or global setting to categorize or filter TLS alert severity before they reach the application layer. To minimize log noise, the recommended strategy is to implement a custom message callback using SSL_set_msg_callback to intercept and filter alerts based on their severity level before they are passed to your logging subsystem.

Implementation Strategy

Because OpenSSL treats alerts as part of the record layer, you must manually inspect the alert's metadata within the callback function. The goal is to distinguish between Warning (non-fatal) and Fatal alerts.

Steps to Filter Alerts

1. Register the Callback: Use SSL_set_msg_callback(ssl, your_callback_function) to hook into the message processing flow.
2. Identify Alert Type: Inside the callback, check if the message type is a TLS alert.
3. Inspect Severity: Examine the first byte of the alert data. In the TLS protocol, the alert level is typically defined as:
   • 1: Warning (non-fatal)
   • 2: Fatal
4. Conditional Logging: Implement logic to immediately return or discard the event if the level is 1 (Warning), while ensuring any level 2 (Fatal) alert is passed to your security logs.

Analysis of Design Trade-offs

While this approach reduces log exhaustion, it introduces a risk of masking protocol anomalies. Specifically, filtering close_notify alerts—which are non-fatal—can make it difficult to diagnose improper connection shutdowns or detect potential truncation attacks where a connection is closed prematurely without a proper TLS shutdown handshake.

Verification Method

To verify the filter is working without compromising security, perform the following tests:

• Non-Fatal Test: Trigger a close_notify alert; verify that the callback intercepts the message and the application logs remain empty.
• Fatal Test: Trigger a handshake failure (e.g., using an unsupported cipher suite); verify that the callback identifies the fatal severity and the event is logged.

Missing Diagnostic Detail: Are you using a specific wrapper library (e.g., PyOpenSSL, BoringSSL) or calling the C API directly? This may change how the callback is registered and how the raw buffer is accessed.