Keystone UUID token validation latency limits under concurrent load

Direct Answer to the Question

1. UUID token latency scaling
UUID token validation does not increase linearly with the number of concurrent requests. The latency remains relatively flat until a specific concurrency threshold is reached. At that knee point, lock contention on the token table and Galera write‑set certification cause a sharp rise in response time.

2. Fernet vs Cached UUID under peak load
When the database is the bottleneck, Fernet tokens exhibit lower and more predictable latency than UUID tokens that rely on a caching layer. Fernet validation is CPU‑bound (AES + HMAC) and stays flat until the Keystone host’s CPU cores approach saturation. Cached UUID tokens can match Fernet latency only if the cache hit ratio stays high and the network to the cache remains low; otherwise, cache misses push them toward the same DB‑related spike.

Confirmed Facts (from research)

• UUID validation requires a DB read for every request; Galera causes non‑linear latency growth beyond a concurrency knee.
• Memcached/Redis caching keeps latency flat until cache miss rates rise or network limits are hit.
• Fernet validation is CPU‑bound and remains stable until CPU cores are fully utilized.
• At peak request rates, Fernet tokens usually outperform cached UUID tokens when the DB is the limiting factor, but they sacrifice instant revocation.

Likely Explanations (contextual)

• Cache miss spikes often happen when the token churn rate exceeds the cache’s write throughput, not because the DB is saturated.
• Galera’s certification phase can become a bottleneck if many nodes simultaneously validate tokens, leading to lock contention that is not purely linear.

Steps to Verify in Your Environment

1. Baseline Benchmark: Deploy Keystone with UUID tokens and a memcached/Redis cache. Run a token‑validation load test (e.g., openstack token issue/validate) while ramping concurrent clients from 10 to 2000. Record average latency, DB CPU, and lock‑wait statistics.
2. Cache‑Hit Monitoring: Ensure the cache hit ratio remains above 95 %. If it drops, attribute latency increases to cache misses rather than DB contention.
3. DB Lock‑Wait Analysis: Inspect Galera’s wsrep_cert_deps and wsrep_cert_latency metrics to pinpoint the concurrency point where latency spikes.
4. Fernet Benchmark: Repeat the test with Fernet tokens enabled (disable UUID provider). Compare latency curves and monitor Keystone host CPU usage.
5. CPU Saturation Check: Verify that CPU utilization stays below ~70 % during the peak of the Fernet test; if it exceeds this, the latency curve will flatten due to CPU limits.

Missing Diagnostic Detail Needed for Recommendation

To decide whether to keep UUID tokens with caching or switch to Fernet, we need to know your current cache hit ratio under production traffic and the Galera certification latency per node. If the cache hit ratio is consistently high and Galera latency is low, UUID tokens may still be acceptable; otherwise, Fernet is likely the better choice.

Practical Take‑away

• Use Fernet tokens if you need predictable, low‑latency validation at high concurrency and can tolerate a configurable token lifetime.
• Keep UUID tokens with a highly tuned cache if immediate revocation is critical and your DB can handle the expected load without hitting the Galera saturation point.