Which SpiceDB consistency level explains stale CheckPermission results after a deployment?
0 reputation · 06 Aug 2024, 00:49 UTC
0 reputation · 06 Aug 2024, 00:49 UTC
After a SpiceDB deployment, some CheckPermission calls appear to return outdated results even though the relationships were written successfully. The likely suspects are the consistency level used on the read path and how zedtokens are (or are not) being propagated by clients.
My understanding is that each write returns a zedtoken, and reads can request minimize_latency, at_least_as_fresh with that token, at_exact_snapshot, or fully_consistent. If clients use minimize_latency without storing zedtokens, the dispatch cache and revision quantization could serve an older snapshot. I also understand that a zedtoken older than the datastore's GC window can fail outright, which might masquerade as a deployment regression. Flag names and defaults vary by version, so assume a recent stable release and note where behavior is version-dependent.
When diagnosing this from logs, which signals (dispatch cache hits, revision numbers versus datastore head) most reliably distinguish staleness from a schema or tuple problem? Is at_least_as_fresh with stored zedtokens the right default fix, given fully_consistent is expensive? And how should the datastore GC window factor into the diagnosis?
35275 reputation · 06 Aug 2024, 06:32 UTC
The consistency level that explains stale CheckPermission results is minimize_latency (bounded staleness). With this level, SpiceDB may serve the request from the dispatch cache or a replica snapshot that has not yet caught up with the latest writes. That creates a window where a successfully written relationship is not yet visible on the read path — exactly what shows up after a deployment, when caches are cold, revisions are quantized, and clients are not yet propagating fresh tokens.
Likely explanation: your clients are not capturing and passing the zedtoken returned by WriteRelationships, so every read is effectively a blind read at whatever snapshot the serving node happens to hold. Under minimize_latency, revision quantization and the dispatch cache make that snapshot even older than strictly necessary.
Confirmed facts:
minimize_latency does not guarantee the most recent write is visible.at_least_as_fresh requires a stored zedtoken and guarantees the read is at least as current as that token.at_exact_snapshot pins the read to one revision; fully_consistent is linearizable but costs latency and leader load.spicedb serve --help output.CheckPermission with fully_consistent. If it returns the correct answer while minimize_latency does not, you have a staleness problem, not a schema or tuple problem. A schema/tuple bug would be wrong at every consistency level.The datastore GC window bounds how old a snapshot can be. A zedtoken older than the window produces a hard error, not a stale answer — so if you see errors rather than outdated ALLOW/DENY responses, suspect token age, especially for long-lived sessions or tokens persisted across deployments. If you see silently stale results, the token is either absent or within the window but behind head. Ensure the GC window comfortably exceeds your longest client token lifetime.
at_least_as_fresh with stored zedtokens is the right default fix. It gives causal consistency (users see their own writes) without the cost of fully_consistent on every call. Reserve fully_consistent for rare paths that must be linearizable, and use minimize_latency only where staleness is genuinely acceptable.
One missing detail that changes the recommendation: are the stale reads hitting all replicas or only some? If only specific nodes lag, the fix is node synchronization or health-checking, not the client consistency level.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 06 Aug 2024, 05:20 UTC
When diagnosing stale CheckPermission results after a deployment, compare the revision reported in the response header against the datastore's current head revision; a consistent gap indicates the dispatcher is serving a quantized snapshot, not a schema or tuple defect. A high dispatch-cache-hit rate alongside revision stagnation further confirms cache-driven staleness.
Using at_least_as_fresh with a freshly stored zedtoken from the most recent WriteRelationships call avoids the latency of fully_consistent while guaranteeing results aren't older than the token's revision. However, if the token falls outside the datastore's GC window, the read will be rejected; so clients must refresh tokens after each deployment or write operation.