Choosing Between AES-GCM and AES-CBC in OpenSSL

The Security Gap: Confidentiality vs. Authenticity

When implementing symmetric encryption in OpenSSL, the primary decision is whether you need only confidentiality (hiding the data) or both confidentiality and authenticity (ensuring the data hasn't been tampered with). Choosing a mode that provides only confidentiality, such as AES-CBC, leaves your application vulnerable to padding oracle attacks unless you manually implement a Message Authentication Code (MAC).

The takeaway: Use AES-GCM for almost all modern applications. It is an Authenticated Encryption with Associated Data (AEAD) mode that handles both encryption and integrity in a single pass, reducing implementation errors and increasing performance on modern CPUs.

Comparison of AES Modes

Engineering Trade-offs

The GCM Nonce Constraint

AES-GCM is highly efficient but fragile regarding its Initialization Vector (IV), referred to as the nonce. If you encrypt two different messages with the same key and the same nonce, an attacker can recover the authentication key and potentially decrypt other messages. You must ensure a unique nonce for every single operation. A common strategy is using a 96-bit random nonce or a strictly incrementing counter.

The CBC Padding Problem

AES-CBC requires the plaintext to be a multiple of the block size (16 bytes). OpenSSL handles this via padding, but this mechanism is the root of "padding oracle attacks," where an attacker can deduce the plaintext by observing how the system responds to incorrectly padded ciphertexts. To secure CBC, you must use an "Encrypt-then-MAC" approach: encrypt the data, then calculate an HMAC (Hash-based Message Authentication Code) over the ciphertext and the IV.

Practical Validation with OpenSSL

You can verify the behavior and performance of these modes using the OpenSSL CLI. These examples assume OpenSSL 1.1.1 or 3.x is installed.

Performance Benchmarking

Run the following commands to see how your hardware handles the two modes. Run these as a standard user on your target deployment environment:

# Test GCM performance
openssl speed -evp aes-256-gcm

# Test CBC performance
openssl speed -evp aes-256-cbc

Expected Result: On CPUs with AES-NI and CLMUL instructions, GCM typically shows higher throughput because the authentication tag is calculated in parallel with the encryption.

Encryption Comparison

Observe how GCM and CBC handle file sizes. Create a small file that is not a multiple of 16 bytes:

echo "ReadMeFeed" > test.txt # 10 bytes

Encrypt using CBC:

openssl enc -aes-256-cbc -salt -in test.txt -out test_cbc.bin -k mysecretpassword

Encrypt using GCM:

openssl enc -aes-256-gcm -salt -in test.txt -out test_gcm.bin -k mysecretpassword

Verification: Check the file sizes using ls -l. The CBC output will be larger than the GCM output (excluding the salt/header) because CBC must pad the 10-byte input up to the 16-byte block boundary.

Limitations and Risks

• GCM Nonce Reuse: Never hardcode a nonce. If you cannot guarantee a unique nonce per key, rotate the key frequently.
• CBC without MAC: Do not use -aes-256-cbc for network protocols unless you have a separate layer providing integrity checks.
• Key Management: Neither mode protects the key itself; ensure keys are stored in a secure vault or HSM (Hardware Security Module) rather than in source code.