Architecture Note: Deploying HashiCorp Vault Transit as Encryption‑as‑a‑Service

Architecture Note: Vault Transit as Encryption‑as‑a‑Service

This note outlines the requirements, minimal design, trust boundaries, operational checks, failure modes, and conditions that would cause a redesign when deploying HashiCorp Vault’s Transit secrets engine as a dedicated encryption‑as‑a‑service layer for application data.

Requirements

• Applications must reach Vault over TLS; mutual TLS is recommended to verify both server and client identities.
• Clients need a token or AppRole that grants read and write capabilities on a specific transit path, e.g. transit/keys//*. The policy should avoid granting sudo or broad capabilities on the entire transit/ mount.
• Applications must base64‑encode plaintext before sending it to transit/encrypt and decode the base64 ciphertext returned by transit/decrypt. The ciphertext is the only data that leaves Vault.
• Vault must run in a highly available mode. The smallest viable HA deployment is a three‑node cluster using Integrated Storage (Raft) or an external highly available storage backend such as Consul.
• Token lifecycle management: applications should retrieve short‑lived tokens via AppRole (role_id/secret_id) or use Kubernetes auth method, and renew tokens before expiry to avoid authentication failures.

Smallest Suitable Design

A three‑node Vault cluster with the Transit engine enabled at mount point transit/. For each distinct data classification create a named encryption key:

vault secrets enable transit
vault write -f transit/keys/pii_key
vault write -f transit/keys/payment_key

Applications wrap the encrypt/decrypt calls in a thin SDK or sidecar that handles TLS, token retrieval (AppRole or Kubernetes), base64 conversion, and retry logic. No key material ever leaves the Vault storage backend; the application only sees ciphertext.

The cluster should be configured with auto‑unseal (e.g., using cloud KMS or Shamir shares stored securely) or a documented manual unseal process with at least two operators.

Trust and Data Boundaries

• Vault is the trust root for the encryption keys; keys never leave the encrypted storage backend.
• Application servers are treated as untrusted clients: they receive only ciphertext and never see plaintext keys.
• Network segmentation places the Vault cluster in a private subnet reachable only from application subnets via TLS on port 8200. Public internet access to Vault is blocked by firewall rules.
• Storage backend data (including the transit key material) is encrypted at rest using Vault’s internal barrier; if using an external backend, ensure that backend also provides encryption at rest.
• Audit devices (file, syslog, or socket) capture every encrypt/decrypt request, recording the token/entity ID, timestamp, and key version used.

Operational Checks

1. Seal status: run vault status on each node or query the telemetry endpoint (/v1/sys/seal-status) and alert if any node reports sealed: true.
2. Replication health (if using performance replicas): monitor vault read sys/replication/status for drift or lag; set alerts on secondary nodes falling behind the primary by more than a configurable threshold.
3. Audit log: enable a file audit device (vault audit enable file file_path=/var/log/vault_audit.log) and periodically verify that encrypt/decrypt entries contain the expected user_id or entity_id and that the path matches the intended transit mount.
4. Key version usage: read the key metadata (vault read transit/keys/) to see the latest version and the min_decryption_version. Ensure older versions are retained until a re‑encryption job confirms all data is encrypted with the newest version.
5. Latency and error rates: instrument the application sidecar to record HTTP response times and status codes from Transit endpoints; trigger alerts on 5xx responses or latencies exceeding your SLA (e.g., >200 ms for 99th percentile).
6. Periodic key rotation: schedule a rotation (vault write -f transit/keys//rotate) and monitor the version increment. Keep previous versions available until a verification step confirms that all ciphertext can be decrypted with the new version.

Failure Modes

• Vault sealed or node loss: encrypt/decrypt calls return connection errors or HTTP 503, causing application encryption/decryption failures. Implement retry with exponential backoff and circuit‑breaker patterns.
• Accidental destruction of a key version (vault delete transit/keys//versions/) renders any ciphertext created with that version unrecoverable. Protect the delete capability with strict policies and consider enabling protection mode on keys.
• Network partition between application and Vault increases latency and may cause timeouts; tune client‑side timeouts and use a sidecar that buffers requests briefly.
• Excessive token policies: if a token is granted sudo or broad read/write on the entire transit/ mount, a compromised token could encrypt or decrypt unintended data. Enforce least‑privilege and audit token capabilities regularly.
• Loss of unseal shares or auto‑unseal credentials results in permanent loss of access to all transit‑encrypted data. Backup Shamir shares according to your recovery plan and test restoration in an isolated environment.

Conditions That Would Change the Design

• Air‑gapped or offline environments: the reliance on a network‑reachable Vault would shift to a local KMS or HSM that can operate without external connectivity.
• High‑throughput bulk encryption (e.g., encrypting large files or streams): the per‑request overhead of Transit may become a bottleneck. Consider envelope encryption where a data key is generated locally, encrypted by Transit, and the data key is cached for bulk operations, or use a dedicated HSM with batch API.
• Regulatory mandates for key custody: if regulations require that keys never leave a specific jurisdiction or be managed by a cloud provider, integrate an external HSM or cloud KMS and use Vault as a broker rather than the root of trust.
• Need for sub‑millisecond latency: enable Transit auto‑tuning (vault write transit/config/ auto_rotate_period="24h" available in Vault 1.12+) or place a caching layer for data keys near the application to reduce round‑trips.
• FIPS 140‑2 validation requirements: if the deployment must run inside a FIPS‑validated module, consider using an external FIPS‑validated HSM and configure Vault to forward transit operations to that HSM via the transit seal wrapper or external plugin.

Practical Verification Steps

1. Enable Transit (requires a token with sudo on sys/mounts):

   vault secrets enable transit

2. Create a key (requires sudo on transit/keys):

   vault write -f transit/keys/example_key

3. Test encrypt/decrypt (run from a host with the Vault CLI and a token that has read and write on transit/keys/example_key):
   # encrypt
ENC=$(vault write -format=json transit/encrypt/example_key plaintext=$(echo -n "test data" | base64) | jq -r .data.ciphertext)
# decrypt
vault write -format=json transit/decrypt/example_key ciphertext="$ENC" | jq -r .data.plaintext | base64 --decode
4. Check audit log: after enabling a file audit device, verify the request appears:

   grep example_key /var/log/vault_audit.log

   Ensure the user_id matches the token/AppRole used.
5. Simulate a node failure: stop one Vault node (systemctl stop vault on that host) or seal the cluster (vault operator seal). Confirm that the encrypt call returns a non‑200 HTTP status or connection error. After restarting the node or unsealing (vault operator unseal), repeat the encrypt/decrypt test and verify success.
6. Verify key version retention: read the key (vault read transit/keys/example_key) and confirm that min_decryption_version is set to the oldest version you intend to keep. After rotating (vault write -f transit/keys/example_key/rotate), check that the version increments and that the previous version is still listed.

Limitations and How to Check Results

• Transit adds network latency; measure round‑trip time with curl -w "%{time_total}\n" -s -o /dev/null https://vault.example.com:8200/v1/transit/encrypt/example_key and compare against your latency budget.
• Key rotation requires retaining old versions; you can verify retention by reading the key (vault read transit/keys/example_key) and ensuring the min_decryption_version has not been advanced prematurely.
• Improper token policies can be detected by periodically checking token capabilities: vault token capabilities -token transit/encrypt/example_key should return only [read update] (or equivalent) and not sudo or deny.
• Loss of unseal shares/shards is irreversible; regularly back up Shamir shares according to your backup policy and test restoration in an isolated environment.

By following the requirements, minimal design, and checks outlined above, you can operate Vault Transit as a reliable encryption‑as‑a‑service layer while understanding the limits that would necessitate a redesign.