Additive schema migration with cache staleness vs. disabling the dispatcher cache during a SpiceDB cutover

Short answer

For a small, low-write application, temporarily disabling the dispatcher cache is the safer trade-off. Set --dispatch-cache-max-staleness=0 (or the equivalent config value) on every node before the schema push, run the additive migration, switch reads, and restore the original staleness setting afterward. The cost is bounded and measurable — higher per-check latency and more datastore queries — while the cost of accepting the staleness window is a brief, hard-to-detect period where authorization decisions contradict your intended state.

Why staleness is the worse risk here

The confirmed mechanics, as you describe them: dispatcher cache entries are keyed by schema version, so a schema push logically invalidates them, but in-flight CheckPermission calls can still be served from pre-invalidation entries for up to dispatcher_cache_max_staleness (10s by default). There is no documented cache-invalidation hook tied to schema migrations, and no supported way to scope invalidation to specific namespaces — invalidation behavior is cluster-wide.

The problem with "just accept 10 seconds" is not the duration; it's the shape of the failure. During the window, some checks resolve against the old schema and some against the new one, split across nodes and cache states you can't observe from the client. For a low-traffic app, the absolute latency cost of disabling the cache is small precisely because your check volume is small. You're buying determinism at a discount.

What pausing writes does and doesn't guarantee

Pausing writes and waiting out the TTL does not guarantee all nodes have dropped old entries. It guarantees only that no new writes reference the old schema. In-flight checks that hit stale cache entries during the TTL window still resolve against the old state, and you have no cross-node signal confirming every node's cache has aged out. Treat "pause writes + wait" as a useful complement to disabling the cache, not a substitute for it.

You're also correct that a zedtoken guarantees consistency at a revision but does not bridge a schema-version transition — it can't help you here.

Recommended cutover sequence

1. Reconfigure all nodes first. Set max staleness to 0 cluster-wide and confirm every node picked it up (check each node's config/debug endpoint or startup logs). A single node left at 10s reintroduces the exact window you're avoiding.
2. Push the additive schema (new relations only).
3. Backfill with batched WriteRelationships, capturing a zedtoken from the final batch so your read switch can require at-least-that revision.
4. Switch reads to the new relations.
5. Restore the original staleness setting once the new schema is fully live, then verify cache hits resume.
6. Drop old relations in a later schema push, ideally with the cache disabled again for that push since it's another schema-version transition.

Verification

Watch the dispatcher cache metrics (hits, misses, evictions) before, during, and after: expect hits to drop to zero while staleness is 0 and recover after you restore it. Rehearse the full sequence in staging with the same cache configuration as production, and compare check latency with the cache on vs. off so the production latency bump holds no surprises. One caveat worth flagging: exact flag names and available debug endpoints vary by version, so confirm them against your deployed v1.30+ build rather than assuming they match this write-up.