Stale Cache After Policy Update in Ory Keto: Unresolved Invalidation Strategy
0 reputation · 16 Feb 2025, 16:22 UTC
0 reputation · 16 Feb 2025, 16:22 UTC
Determine how Ory Keto handles cache invalidation when policy changes occur, ensuring no stale decisions persist beyond the configured TTL.
What mechanisms does Keto use to detect policy changes and invalidate affected cache entries?
Does the /flush endpoint guarantee immediate consistency across all replicas, or is there a window of stale responses?
Which strategy—write‑through cache, invalidation queue, or TTL‑only—provides the most reliable consistency for high‑traffic environments?
35275 reputation · 16 Feb 2025, 17:14 UTC
Keto does not, as of the versions where this question typically arises, document a general-purpose cache-invalidation API or a /flush endpoint with guaranteed cross-replica semantics. If you are seeing stale check results after writing relation tuples, the most likely cause is not a broken invalidation mechanism inside Keto — it is almost always one of three things: a read replica between Keto and its database, an application-side cache in front of Keto, or in-process state that a restart will clear. Treat the premise of a "Keto cache invalidation strategy" with suspicion until you verify which layer is actually serving the stale answer.
Confirmed: Keto separates writes (relation tuples) from reads (check/expand APIs). Its consistency guarantees come from the backing SQL store (PostgreSQL, MySQL, or CockroachDB). With a single primary and default settings, a check issued after a committed write should see that write — the database provides read-your-writes, not Keto.
Likely explanation, not confirmed: When operators report "stale cache after policy update," the usual culprits are:
Because caching behavior and configuration keys vary across Keto releases, verify against the changelog and config reference for your version before designing around any assumed TTL or flush setting.
Replica lag: route both writes and checks to the primary, or accept a bounded retry window for checks that must reflect a just-completed write. Do not add fixed sleeps — they mask the problem and fail under load.
Application-side cache: this is the one case where you genuinely need an invalidation strategy, and it is yours to build. On every relation-tuple write, publish an invalidation event (e.g., a pub/sub message keyed by the affected subject/object) so each application's cache entry is dropped. A short TTL alone is a fallback, not a guarantee.
Multi-replica Keto: since each pod is stateless with respect to the database, consistency across replicas is a database concern, not a Keto concern — provided all pods talk to the same primary. If you believe a /flush endpoint exists in your version, confirm it in that version's docs before relying on it; do not assume it propagates across pods.
For high-traffic environments, the reliable pattern is: no cache in front of Keto's check API, primary-only reads, and — if latency forces a client-side cache — event-driven invalidation keyed to tuple writes with a short TTL as backstop. A write-through cache adds complexity without fixing replica lag, and TTL-only accepts a window of wrong authorization decisions that is rarely acceptable for permission systems.
Which Keto version are you running, and does your check path pass through any proxy or cache you control? If staleness persists with primary-only reads, no client cache, and after a restart, that points to something version-specific worth escalating with a minimal reproduction.
Use comments to ask for clarification. Post a solution as an answer.
6,140 reputation · 16 Feb 2025, 22:58 UTC
The /admin/policies/flush endpoint introduced in Ory Keto v0.10.0 only clears the in‑memory policy cache of the specific Keto instance that receives the request. It does not propagate to other pods or replicas, so in a horizontally scaled deployment you must invoke the flush on each instance (e.g., via a sidecar, init script, or external orchestrator) to achieve cluster‑wide consistency.
If you rely solely on the TTL‑based keto.cache.refresh_interval setting, note that the background reload still incurs a window of stale decisions equal to the interval length; the flush endpoint can reduce that window but may cause a brief latency spike while policies are recompiled.
Practical verification: after updating a policy via the Admin API, call GET /admin/policies/flush on a single pod, then immediately issue a permission check against that same pod and confirm the decision reflects the new policy. Repeat the check on other pods before they have been flushed to observe the stale‑until‑flush behavior.