Architecture Note: Using SpiceDB's Watch API for Real‑Time Permission Propagation

Requirements

SpiceDB's Watch API must deliver tuple changes (inserts or deletes) from the underlying store to interested services with sub‑second latency, without requiring an external message broker. Consumers need a reliable way to detect reconnects and handle possible duplicate events. The stream must be protected so that only authorized callers can receive tuple updates.

Minimal Suitable Design

The Watch implementation leans on the database's native change‑data‑capture (CDC) facility:

• PostgreSQL: logical replication slots.
• CockroachDB: CHANGEFEED.
• Each CDC event is translated directly into a WatchResponse protobuf message and sent over a gRPC server‑side streaming RPC.
• No intermediate queue is added; the service merely forwards the CDC payload.

This keeps the path short: write → WAL → CDC slot → SpiceDB Watch handler → gRPC stream.

Trust and Data Boundaries

The Watch endpoint is exposed inside the service mesh and protected by mutual TLS. Authorization is performed at the gRPC level: a token must contain the watch permission on the root namespace. Because the stream originates from SpiceDB itself, there is no risk of injection; the only trust boundary is the client‑to‑SpiceDB connection.

Operational Checks

To verify that the Watch path is healthy, monitor the following:

1. Watch lag – metric spicedb_watch_lag_seconds (time between a tuple commit and its emission on the stream). Set an alert if the 95th‑percentile exceeds, for example, 500 ms under expected load.
2. Reconnect rate – count of gRPC stream resets per minute. A steady increase may indicate network instability or back‑pressure.
3. Outbound queue depth – internal metric showing how many WatchResponses are waiting to be sent. Sustained growth signals that the consumer cannot keep up.

These metrics are exported via Prometheus when SpiceDB is started with the --metrics-prometheus-addr flag.

Failure Modes and Handling

• Network‑induced disconnections – the gRPC layer returns UNAVAILABLE. SpiceDB automatically attempts reconnection with exponential back‑off (starting at 100 ms, capped at 10 s). Consumers should treat any non‑OK status as a signal to reconnect.
• Schema change incompatibility – if a relation definition is altered while a stream is open, decoding may fail, causing the stream to close with UNIMPLEMENTED or INTERNAL. After the schema version is updated, the client can reconnect and resume.
• CDC pipeline stall – replication slot exhaustion (PostgreSQL) or CHANGEFEED pause (CockroachDB) leads to growing lag and eventually a RESOURCE_EXHAUSTED status. Monitoring slot usage or feed lag helps detect this before it impacts Watch.

All failure conditions are surfaced as gRPC status codes; they are not fatal to the SpiceDB process.

Conditions That Would Change the Design

Consider revising the minimal design if any of the following become true:

• Required delivery semantics shift to exactly‑once; then an idempotency layer or checkpoint store (e.g., Redis) would be needed.
• Write throughput exceeds the capacity of a single CDC slot, causing persistent lag despite consumer scaling. In that case, sharding tuples by namespace or introducing a lightweight broker (e.g., NATS) between SpiceDB and consumers could be warranted.
• Regulatory constraints demand that tuple changes never leave a specific trust zone; then the Watch endpoint would need to be terminated inside that zone with strict mTLS and no external exposure.

Practical Verification Steps

These steps illustrate how to confirm the Watch API behaves as described. Run them in a test environment; do not rely on them for production SLAs.

1. Start SpiceDB (requires Docker read access to the image):

   docker run -d --name spicedb -p 50051:50051 -p 9090:9090 \
     authzed/spicedb:latest \
     --grpc-addr=:50051 \
     --metrics-prometheus-addr=:9090

2. Initialize a namespace and relation (run inside the container or via the CLI with appropriate permissions):

   zed write --token=<admin-token> \
     "user:alice#member@tenant:acme" \
     "tenant:acme#admin@role:admin"

3. Watch for changes in a separate terminal (requires the same token with the watch permission):

   zed watch --token=<watch-token> \
     "user:*#member@tenant:*"

4. Commit a new tuple (e.g., add Bob as a member):

   zed write --token=<admin-token> \
     "user:bob#member@tenant:acme"

5. Observe the Watch output – you should see a JSON‑encoded TupleMutation with the inserted tuple appear within a few seconds. No extra output is guaranteed; the exact format follows the protobuf definition.
6. Check lag – while the watch is running, query Prometheus:

   curl http://localhost:9090/metrics | grep spicedb_watch_lag_seconds

   Verify the reported value stays under your threshold (e.g., < 0.5 s) during steady writes.
7. Test a schema change – add a new relation, then watch the stream:

   zed write --token=<admin-token> \
     "definition role:admin { permission view -> user:alice }"

   Existing watch streams should close with a non‑OK status; after the schema is updated, reconnect the watch and verify it resumes without manual intervention.

Limitations

The Watch API does not guarantee exactly‑once delivery; during reconnects a client may receive duplicate WatchResponse messages. Consumers must either be idempotent with respect to tuple updates or persist a checkpoint (e.g., the last known commit timestamp) and discard events that are older than that checkpoint.

High write volume can saturate the underlying changefeed, increasing lag. If the lag metric consistently approaches the alert threshold, consider either scaling the SpiceDB instance (more CPU/I/O for the CDC source) or sharding the tuple space across multiple SpiceDB clusters and fanning out watches accordingly.

Summary

SpiceDB's Watch API provides a low‑latency, broker‑free permission change stream by directly coupling the database's CDC mechanism to a gRPC server‑side RPC. Trust is enforced via mTLS and the watch permission, operational health is measured through lag, reconnect rate, and outbound queue metrics, and failure modes are handled automatically with exponential back‑off and clear status codes. The design remains suitable as long as exactly‑once guarantees are not required and write throughput stays within the capacity of a single changefeed; otherwise, consider sharding or introducing a lightweight broker.