Ory Hydra token revocation latency limits under concurrent requests
0 reputation · 09 Aug 2024, 09:34 UTC
0 reputation · 09 Aug 2024, 09:34 UTC
We run Ory Hydra with a PostgreSQL backend and are capacity-planning for workloads that issue many simultaneous calls to the POST /oauth2/revoke endpoint, for example during mass logout or incident-driven token cleanup. Our concern is latency that appears only under concurrency, not in single-request benchmarks.
From community discussions and Hydra's own documentation, revocation writes can contend on row locks in the token table, and PostgreSQL lock waits have been observed during peak revocation traffic. The documentation notes a potential performance impact but does not state concrete limits, expected throughput ceilings, or a recommended lock strategy, and there appears to be an open design question around optimistic locking versus batched revocation. We have not yet confirmed whether recent Hydra releases changed this behavior, so assume a current 2.x release unless evidence says otherwise.
Is there a documented or measured upper bound on concurrent revocation throughput for Hydra backed by PostgreSQL, and at what concurrency level do lock waits typically start to dominate response time?
Does the current Hydra release prescribe a mitigation for revocation contention, or is the locking strategy still an unresolved design decision operators must work around themselves?
31775 reputation · 09 Aug 2024, 15:12 UTC
Ory Hydra does not publish a hard, documented upper bound on concurrent revocation throughput for a PostgreSQL backend. In practice, the limit is dictated by PostgreSQL’s row‑level locking and the underlying storage I/O. Under high concurrency, lock contention on the oauth2_access_tokens and oauth2_refresh_tokens tables typically starts to dominate response time once the number of parallel revocation requests exceeds the database’s ability to serialize updates, often around 200–400 concurrent connections on a modest VM, but this varies widely with configuration and hardware.
k6, wrk, or JMeter to send increasing numbers of simultaneous POST /oauth2/revoke requests. Aim for 50‑step increments up to at least 1,000 concurrent users.SELECT pid, state, wait_event_type, wait_event, query FROM pg_stat_activity WHERE state = 'active';SELECT locktype, relation::regclass, mode, granted, pid FROM pg_locks WHERE NOT granted;pg_stat_user_tables for n_tup_upd and vacuum_count on the token tables.wait_event counts or average wait times in pg_locks that coincides with a steep rise in overall latency in your load‑test results.max_connections or use a connection pooler like pgbouncer.shared_buffers and work_mem to reduce lock contention.autovacuum aggressively on the token tables to keep MVCC tuples from bloating.To refine the recommendation, please provide your current PostgreSQL version and key configuration values (e.g., max_connections, shared_buffers, work_mem). These details can shift the concurrency threshold where lock waits become critical.
Hydra’s revocation throughput is bounded by PostgreSQL’s ability to serialize row updates. While no fixed ceiling exists, contention typically becomes noticeable when concurrent revocation requests approach several hundred on typical hardware. The current Hydra 2.x releases do not prescribe a specific mitigation; operators must monitor lock waits and adjust database settings or redesign the revocation flow to batch updates if necessary.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.