EVP_PKEY_verify() heap leak during signature verification failure

Addressing Memory Exhaustion in OpenSSL 3.x

In OpenSSL 3.x, EVP_PKEY_verify() returns 1 for success, 0 for a verification failure (invalid signature), and -1 for a systemic error. Regardless of which return code is received, the caller is responsible for releasing the EVP_PKEY_CTX and the EVP_PKEY. Memory exhaustion during failure typically occurs when the cleanup sequence is bypassed or when provider-level allocations are not unwound during early exits.

Likely Leak Vectors in Provider Architecture

While specific line-level exit paths vary by version, leaks in the 3.x provider architecture generally stem from two areas:

• Partial ASN.1 Construction: When decoding malformed signatures or keys, the engine may allocate nested ASN.1 structures (such as AlgorithmIdentifier). If an error occurs after a child node is allocated but before the parent structure is fully formed or ownership is transferred to a managed object, the child may remain orphaned in the heap.
• Provider-Native Temporaries: Providers may allocate temporary buffers for digests or decoded key parameters. If the provider's internal verify implementation returns an error code without calling its own internal cleanup routines, those blocks remain allocated until the provider itself is unloaded.

Recommended Cleanup Pattern

To ensure manual reference counts are cleared and nested structures are freed, follow this strict lifecycle pattern:

1. Reference Management: If the EVP_PKEY is shared across multiple contexts, use EVP_PKEY_up_ref() to increment the reference count.
2. Unified Error Path: Treat both 0 (verification failure) and -1 (system error) as triggers for the same cleanup sequence.
3. Ordered Deallocation: Free the context before the key to avoid potential use-after-free scenarios within the provider.

/* Safe Cleanup Pattern */
int result = EVP_PKEY_verify(ctx, sig, siglen, tbs, tbslen);

// Always execute regardless of result (1, 0, or -1)
EVP_PKEY_CTX_free(ctx);
EVP_PKEY_free(pkey);
// Free any caller-allocated ASN.1 or MD_CTX objects here

Verification and Diagnostics

Because OpenSSL uses OPENSSL_malloc, standard heap profilers may miss these leaks. Use the following methods for verification:

• Internal Tracking: Call CRYPTO_mem_leaks() in a test harness after the cleanup sequence to detect leaked blocks.
• External Tooling: Run the application under AddressSanitizer (ASan) or Valgrind, ensuring you use an interceptor for OPENSSL_malloc.

Diagnostic Requirement: To provide a more specific exit-path analysis, please specify the exact OpenSSL 3.x minor version and the specific algorithm (e.g., RSA-PSS, ECDSA) being used, as provider implementations differ significantly.