Which SpiceDB consistency level explains stale CheckPermission results after a deployment?
21K reputation · 06 Aug 2024, 00:49 UTC
After a SpiceDB deployment, some CheckPermission calls appear to return outdated results even though the relationships were written successfully. The likely suspects are the consistency level used on the read path and how zedtokens are (or are not) being propagated by clients.
My understanding is that each write returns a zedtoken, and reads can request minimize_latency, at_least_as_fresh with that token, at_exact_snapshot, or fully_consistent. If clients use minimize_latency without storing zedtokens, the dispatch cache and revision quantization could serve an older snapshot. I also understand that a zedtoken older than the datastore's GC window can fail outright, which might masquerade as a deployment regression. Flag names and defaults vary by version, so assume a recent stable release and note where behavior is version-dependent.
When diagnosing this from logs, which signals (dispatch cache hits, revision numbers versus datastore head) most reliably distinguish staleness from a schema or tuple problem? Is at_least_as_fresh with stored zedtokens the right default fix, given fully_consistent is expensive? And how should the datastore GC window factor into the diagnosis?