Additive schema migration with cache staleness vs. disabling the dispatcher cache during a SpiceDB cutover
0 reputation · 23 Jul 2023, 09:21 UTC
0 reputation · 23 Jul 2023, 09:21 UTC
I'm planning a zero-downtime migration of a small application's authorization data into SpiceDB (assume v1.30+). The intended path is the documented additive pattern: push a new schema version that only adds relations, backfill relationships with batched WriteRelationships calls, switch reads to the new relations, and drop the old relations in a later schema push.
The unresolved point is the dispatcher cache. Because cache entries are keyed by schema version, a schema write invalidates them, but in-flight CheckPermission requests can still be served from pre-invalidation entries for up to dispatcher_cache_max_staleness (10s by default). There appears to be no documented cache-invalidation hook tied to schema migrations, so my options seem to be: accept the staleness window, set dispatcher_cache_max_staleness=0 during the cutover (cluster-wide cache off, higher tail latency and datastore load), or add external coordination such as pausing writes and waiting out the TTL.
A zedtoken from a write guarantees read consistency at that revision, but I understand it does not bridge a schema-version transition.
For a small app with low write volume, which trade-off is safer in practice: accepting up to ~10s of stale checks, or temporarily disabling the cache? Is there any supported way to scope the staleness risk to only the namespaces being migrated? Does pausing writes plus waiting the TTL actually guarantee all nodes have dropped the old entries?
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.