Real-Time Cache Invalidation with SpiceDB Watch Streams

The Cache Invalidation Problem in Distributed Authorization

You've moved authorization decisions to SpiceDB. The CheckPermission calls are fast, but your application still caches results to avoid round trips. Now a user is removed from a group, and your cache serves stale permissions for minutes until TTL expiry. Polling SpiceDB for changes defeats the purpose of centralizing auth. This is the classic cache invalidation problem, and SpiceDB's Watch streams are designed to solve it.

How Watch Streams Work

Watch streams are gRPC subscriptions that push tuple changes to clients in real time. When a WriteRelationships or DeleteRelationships call commits, SpiceDB emits events containing the affected tuples and the operation type. Clients receive these events over a persistent connection, keyed by a filter that specifies which relationships to track.

The filter uses the same tuple syntax as the data model: resource_type, optional relation, and optional subject_filter. A filter of document#viewer@user:* watches all viewer relations on documents for any user. A filter of folder#member@group:engineering#member watches only the engineering group's membership in folders.

Events include the tuple key, operation (TOUCH for create/update, DELETE for removal), and a transaction ID for ordering. The stream guarantees at-least-once delivery; clients must handle duplicates.

Worked Example: Invalidating a Permission Cache

Consider a Go service that caches CheckPermission results in Redis with a 5-minute TTL. We'll add a Watch stream to invalidate entries immediately when relationships change.

Schema Context

definition document {
  relation viewer: user | group#member
  relation editor: user | group#member
  permission view = viewer + editor
  permission edit = editor
}

definition group {
  relation member: user | group#member
}

Watch Client Implementation

package authcache

import (
    "context"
    "log"
    "time"
    
    "github.com/redis/go-redis/v9"
    "github.com/authzed/spicedb/pkg/tuple"
    "github.com/authzed/spicedb/pkg/watch"
    v1 "github.com/authzed/api/v1"
)

func StartCacheInvalidator(ctx context.Context, client v1.PermissionsServiceClient, rdb *redis.Client) error {
    // Watch all viewer and editor relations on documents
    filter := &v1.RelationshipFilter{
        ResourceType: "document",
        OptionalRelation: "viewer", // empty means all relations
    }
    
    req := &v1.WatchRequest{
        Filter: filter,
        // Start from now; use a checkpoint token for resume after restart
    }
    
    stream, err := client.Watch(ctx, req)
    if err != nil {
        return err
    }
    
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
            resp, err := stream.Recv()
            if err != nil {
                log.Printf("watch stream error: %v", err)
                // Implement exponential backoff and reconnect
                return err
            }
            
            for _, update := range resp.Updates {
                // Build cache key pattern: perm:document:{id}:view:{user_id}
                // Invalidate all users who might be affected
                invalidateCacheForTuple(rdb, update.Relationship)
            }
        }
    }
}

func invalidateCacheForTuple(rdb *redis.Client, rel *v1.Relationship) {
    // Extract resource ID and subject
    resourceID := rel.Resource.ObjectId
    subject := rel.Subject
    
    // Invalidate both view and edit permissions for this subject on this resource
    // Pattern: perm:document:{resourceID}:view:{subjectID}
    pattern := fmt.Sprintf("perm:document:%s:*:%s:%s", resourceID, subject.ObjectType, subject.ObjectId)
    
    // Use SCAN to avoid blocking Redis
    iter := rdb.Scan(ctx, 0, pattern, 100).Iterator()
    for iter.Next(ctx) {
        rdb.Del(ctx, iter.Val())
    }
    if err := iter.Err(); err != nil {
        log.Printf("cache invalidation scan error: %v", err)
    }
}

Running the Watcher

Deploy this as a sidecar or separate worker process with the same SpiceDB credentials as your API servers. Required permissions: watch permission on the SpiceDB instance (typically granted via a dedicated service account with the watch role). The watcher must maintain a persistent gRPC connection; configure your load balancer for long-lived connections (disable idle timeouts).

To resume after restarts, store the last received transaction ID (from resp.Checkpoint) and pass it in a subsequent WatchRequest with StartCheckpoint. This avoids reprocessing old events.

Trade-offs and Limitations

• At-least-once delivery: Your invalidation logic must be idempotent. Deleting a Redis key twice is harmless, but if you're updating a version counter, use atomic operations.
• Ordering within a transaction: Events from a single write are ordered, but events across concurrent transactions may interleave. The checkpoint token provides a total order for resume.
• Filter granularity: Broad filters (e.g., all relations on all resources) increase event volume. Narrow filters reduce noise but require multiple streams for full coverage. Balance based on your tuple cardinality.
• No built-in retry: The gRPC stream fails on network partitions. Implement reconnection with exponential backoff and checkpoint resume. SpiceDB retains history for a configurable window (default 24 hours); if your client is down longer, you'll need a full cache flush.
• Consistency window: There's a brief period between a write committing and the watch event arriving. For strict consistency, combine Watch with a short TTL (e.g., 30 seconds) as a safety net.

Verifying the Integration

Test the invalidation path without SpiceDB by publishing synthetic events to a test Redis instance. For end-to-end verification:

1. Start SpiceDB with spicedb serve --datastore-engine=memory for local testing.
2. Write a test tuple: spicedb write document:readme#viewer@user:alice.
3. Prime the cache: call your CheckPermission wrapper for user:alice on document:readme with view.
4. Delete the tuple: spicedb delete document:readme#viewer@user:alice.
5. Confirm the cache key is deleted within milliseconds (check Redis MONITOR or log output).
6. Verify a subsequent CheckPermission call returns PERMISSIONSHIP_NO_PERMISSION and repopulates the cache correctly.

For production validation, instrument the watcher with metrics: events received, invalidations performed, reconnection count, and end-to-end latency from write to cache deletion. Alert on reconnection storms or event processing lag exceeding your consistency SLA.

Next Steps

If your cache invalidation currently relies on TTL alone, add a Watch stream for your highest-traffic permission checks. Start with a narrow filter on the most security-sensitive relation (e.g., document#editor). Measure the reduction in stale-permission incidents and the event volume before expanding. The spicedb watch CLI command (spicedb watch document#viewer@user:*) is useful for manual inspection of event flow during development.