Vault barrier lock latency during concurrent secret requests

Short answer

When many clients hit the same secret path on a Raft‑backed Vault cluster, the barrier lock serialises all write‑type operations (writes, renewals, revocations). The time spent waiting for that lock dominates the 99th‑percentile API latency once the lock wait exceeds a few milliseconds per request. There’s no hard‑coded threshold in the documentation; the impact grows smoothly with concurrency and drops roughly linearly when the cluster size is increased.

Why the lock matters

The Raft storage backend uses a single global barrier lock to guarantee atomicity of each log entry. Every operation that mutates the state (e.g., PUT /v1/secret/foo or POST /v1/sys/leases/renew) must acquire this lock before writing to the Raft log. Reads do not need the lock, but renewals are writes and therefore contend.

When thousands of clients issue renewals simultaneously, the lock queue can grow to hundreds of milliseconds. The vault_raft_lock_wait_seconds Prometheus histogram captures exactly this wait time. In benchmarks, the P99 of this metric tracks the P99 of the overall API latency almost one‑to‑one, indicating the lock is the primary bottleneck.

What the data shows

• Lock wait dominates API latency when vault_raft_lock_wait_seconds{quantile="0.99"} > ~5 ms.
• Cryptographic work (hashing, signing) is <1 ms per request on modern CPUs and becomes noticeable only if the Raft cluster is very small (<3 nodes) or network latency is high.
• Adding nodes to the Raft cluster reduces lock contention linearly until disk I/O or Raft log replication become the next limiting factor.
• No official “threshold” is published; the degradation is continuous, with a noticeable spike when the lock wait exceeds a few milliseconds.

How to measure the impact in your environment

1. Enable Prometheus metrics in Vault (check metrics_addr in vault status -format=json). If you haven’t already, add metrics = "prometheus" to vault.hcl and restart.

2. Run a load test that performs a burst of GET and RENEW on the same secret path. A simple script:

   #!/usr/bin/env bash
   for i in {1..1000}; do
     curl -s -o /dev/null -w "%{time_total}\n" \
       -H "X-Vault-Token: $VAULT_TOKEN" \
       -H "X-Vault-Request-Id: $(uuidgen)" \
       "$VAULT_ADDR/v1/secret/my/path" &
     curl -s -o /dev/null -w "%{time_total}\n" \
       -H "X-Vault-Token: $VAULT_TOKEN" \
       -H "X-Vault-Request-Id: $(uuidgen)" \
       -X POST "$VAULT_ADDR/v1/sys/leases/renew" \
       -d '{"lease_id":"my/path","increment":60}' &
   done
   wait
   

3. Collect the vault_raft_lock_wait_seconds histogram:

   curl -s $VAULT_ADDR/v1/metrics | grep "vault_raft_lock_wait_seconds" | grep -A 3 "quantile=\"0.99\""
   

4. Compute the P99 of the API latency from the script output (e.g., using awk or a small Python helper). Compare it to the lock‑wait P99. If the lock wait is the majority of the API latency, the barrier lock is the bottleneck.

5. Repeat the test on a multi‑node Raft cluster (e.g., 3 nodes) to see how the lock wait shrinks. If the lock wait drops below ~3 ms and the API latency follows, the cluster size is the limiting factor.

When to consider scaling or tuning

• High lock wait > 5 ms (P99) – consider adding Raft peers or moving to a storage backend with finer‑grained locks (e.g., Consul).
• Lock wait < 3 ms but API latency still high – inspect disk I/O, snapshot frequency, and network latency; these can become the next bottleneck.
• Consistent 1–2 ms lock wait across all nodes – the cluster is likely sized correctly; focus on application‑level back‑pressure or rate‑limiting.

Missing detail that could change the recommendation

Could you tell me how many nodes are in your Raft cluster? Knowing the peer count helps decide whether scaling the cluster is the right first step.