Diagnosing HashiCorp Vault Transit Engine Encryption/Decryption Failures

Recognizable condition

When applications interact with the Vault Transit secrets engine you may observe one of the following symptoms:

• Encryption or decryption calls return "permission denied".
• Decryption yields stale or incorrect plaintext after a key rotation.
• Request latency consistently exceeds 500 ms under normal load.
• The Transit engine responds with HTTP 503 Service Unavailable.

Cause / diagnostic table

Symptom	Likely cause	Verification check
Permission denied on encrypt/decrypt	Client token missing required capability (e.g., transit:/encrypt/ or transit:/decrypt/)	Run vault token capabilities <token> transit/encrypt/<key-name> and confirm the output includes "update" (or "read" for decryption).
Stale/incorrect plaintext after rotation	Application using an older key version; Vault returns ciphertext encrypted with the latest version but the app decrypts with a cached older version.	Inspect the ciphertext prefix (e.g., "vault:v1:") and compare with the application’s key‑version cache. Rotate the app or force a refresh of the key‑ring.
High latency (>500 ms)	Transit engine CPU or I/O bottleneck, or excessive request rate approaching the configured max request limit.	Check Prometheus metric vault_transit_request_duration_seconds (average and 95th percentile). Also review vault_transit_requests_total for spikes.
503 Service Unavailable	Transit request rate exceeds the engine’s max request limit (default 1000/s) or the underlying storage backend is throttling.	Look at audit logs for "request rejected" entries and monitor vault_transit_requests_rejected_total.

Ordered checks

1. Validate token capabilities

   Run on a Vault CLI or API with a token that has sudo or root privileges to inspect another token:

   vault token capabilities <client-token> transit/encrypt/<key-name>

   Expected output: a list containing "update". If missing, the policy needs to be updated.

   Where to run: Any Vault client with network access to the Vault server. Permissions: Ability to read token capabilities (usually a token with sudo or root). Risk: None; this is a read‑only operation.

2. Check key version usage

   Encrypt a test value and examine the ciphertext prefix:

   vault write -format=json transit/encrypt/<key-name> plaintext=$(base64 <<< "test")

   Expected ciphertext format: vault:v<version>:.... Note the version number.

   Then attempt decryption with the same token:

   vault write -format=json transit/decrypt/<key-name> ciphertext=<ciphertext-from-previous-step>

   If decryption fails or returns unexpected data, verify the application is not caching an older key version.

   Where to run: Vault CLI. Permissions: Token with transit:/encrypt/<key-name> and transit:/decrypt/<key-name>. Risk: None; uses test data.

3. Measure latency and request rate

   If Prometheus is integrated, query:

   # average latency over 5m
   avg_over_time(vault_transit_request_duration_seconds[5m])
   # 95th percentile
   histogram_quantile(0.95, sum(rate(vault_transit_request_duration_seconds_bucket[5m])) by (le))
   # request rate
   sum(rate(vault_transit_requests_total[5m]))

   Look for values >0.5 s latency or request rates approaching the configured limit (see Vault config max_request_limit).

   Where to run: Prometheus query interface or Grafana. Permissions: Read access to the metrics endpoint. Risk: None.

4. Inspect audit logs for throttling

   Search the enabled audit device for entries where auth is successful but response contains permission_denied or request_rejected:

   grep "transit" /var/log/vault/audit.log | grep "request_rejected"

   If such entries appear, the engine is hitting its max request limit.

   Where to run: On the Vault server or log aggregation system. Permissions: Read access to audit logs. Risk: None.

Fixes tied to findings

• Missing capability: Update the token’s policy (or create a new policy) to include the needed paths, e.g.:

path "transit/encrypt/<key-name>" {
  capabilities = ["update"]
}
path "transit/decrypt/<key-name>" {
  capabilities = ["update"]
}

Then re‑issue or renew the token with the updated policy.

• Stale key version: Ensure the application reloads the Transit key‑ring after a rotation. This can be done by:
• Restarting the service, or
• Calling Vault’s transit/keys/<key-name>/rotate endpoint and then immediately refreshing the cached key version.

After the fix, repeat the encrypt/decrypt test; the ciphertext version should match the latest key version and decryption should succeed.

• High latency / throttling:
• Increase the max_request_limit in the Transit engine tuning parameters (requires a Vault server restart or a sys/tuning call if supported). Example:

vault write sys/tuning/transit max_request_limit=2000

• Scale the Vault cluster (add more performance standbys or increase CPU/RAM on the active nodes).
• Client‑side: implement retry with exponential backoff and consider batching encrypt/decrypt calls where appropriate.
• 503 Service Unavailable due to throttling:
• Apply the same max_request_limit increase as above.
• Monitor vault_transit_requests_rejected_total; it should drop to zero after the limit is raised and traffic stabilizes.

Escalation criteria

If after applying the relevant fix:

• Permission errors persist despite correct policies,
• Latency remains >500 ms after raising the request limit and checking node health, or
• 503 errors continue with audit logs showing request_rejected at a rate >1 % of total traffic,

then escalate to the Vault operations team for deeper investigation of:

• Underlying storage backend performance (e.g., Consul, integrated storage raft latency).
• Possible sealing/unsealing cycles causing transient unavailability.
• Review of Vault audit and telemetry for systemic issues.

Collect the following data before opening a ticket:

• Output of vault status and vault read sys/health.
• Prometheus queries for latency and request rate over the last 15 minutes.
• Relevant audit log snippets (redacted) showing denied or rejected requests.