Question
Measuring ListRoles Latency Under Concurrency and Unresolved Policy Precedence
Rin Maple
0 reputation · 31 Oct 2022, 18:57 UTC
76K views1
Loading…
0 reputation · 31 Oct 2022, 18:57 UTC
Spicedb’s ListRoles gRPC endpoint is a core component whose response time directly impacts system throughput.
The objective is to quantify the latency of this endpoint when handled with high concurrency, using the default page size, to determine whether pagination or caching is required before any policy evaluation optimizations.
Logs consistently report the exact string “error: permission denied” when an unauthorized read occurs, confirming the documented failure condition. Because Spicedb does not define deterministic precedence for overlapping allow and deny statements across multiple role assignments to the same subject, the observed nondeterministic policy evaluation remains an open design question. Which specific metrics should be captured to reliably assess the bottleneck? How does the lack of defined policy precedence affect correctness guarantees? What is the impact of varying concurrency levels on the measured latency?
SpiceDB's ListRoles gRPC endpoint response time directly impacts system throughput. Logs consistently report the string error: permission denied when an unauthorized read occurs, confirming the documented denial path. SpiceDB does not define deterministic precedence for overlapping allow and deny statements across multiple role assignments to the same subject, making policy evaluation nondeterministic when overlapping assignments exist. The effective default page size is version-sensitive and must be confirmed for the deployed instance; it directly changes per-RPC work and latency.
If p50/p95 rise with flat server CPU but rising datastore latency, the bottleneck is likely the datastore or connection pool. If handler wait time and in-flight RPCs grow while datastore latency stays flat, the SpiceDB server is saturated. Growth in p99/p999 without corresponding CPU or datastore changes typically signals queueing, garbage-collection thrashing, or cache contention. Segmenting these metrics by the presence of overlapping allow/deny assignments in the schema isolates whether policy precedence ambiguity contributes to tail-latency inflation.
At low concurrency (e.g., < 20 requests/second), ListRoles latency should remain near the baseline established in a warm-cache, single-request run. As concurrency increases, mean latency climbs once queued RPCs exceed the handler goroutine pool size. Throughput plateaus near the server's maximum request-processing capacity, and error rates rise when the connection pool or datastore session limit is exhausted. p99 and p999 latency tails lengthen notably once concurrency crosses the threshold where request cancellation and context deadlines become common. These patterns hold when concurrency is stepped in controlled increments with a fixed default page size; mixing cold-cache and warm-cache runs or varying page size mid-test confounds datastore latency with server-side queueing.
One missing diagnostic detail that would change the recommendation: what is the effective default page size for the deployed SpiceDB version? Confirming this value is essential before recommending pagination or caching adjustments, as page size directly scales per-RPC work and alters the latency baseline.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.
spicedb