SpiceDB watch stream latency under high write throughput: what limits apply?

Question

We are designing a service that relies on SpiceDB's watch API to propagate authorization changes to downstream caches in near real time. The documented behavior is that the watch stream delivers changes with the consistency guarantees of the backing datastore, but it is less clear what happens to delivery latency when write throughput spikes.

Our concern is subscription queue buildup: if many clients hold open watch streams while the datastore processes a burst of relationship writes, do events queue per-subscription, and is there a documented bound or backpressure mechanism before events are dropped or the stream is terminated? We have not yet run a benchmark, and we want to understand the expected limits before measuring.

Assuming a recent SpiceDB version on PostgreSQL or CockroachDB as the datastore:

Is there a documented per-stream buffer or timeout that governs how far behind a watch consumer can lag before the stream fails?
Does watch latency scale with total write throughput, or only with writes matching the subscribed object types?
Are there recommended metrics to observe queue depth before deciding whether to shard watch consumers?

Tasadduq Burney · Answer

In SpiceDB, the watch stream latency is primarily governed by the database's ability to tail the transaction log and the server's memory-resident buffering for slow consumers. When write throughput spikes, the bottleneck is typically not a fixed-size queue, but rather the resource consumption on the SpiceDB node itself.

Subscription Queueing and Backpressure
SpiceDB does not have a documented, hard per-stream buffer limit that drops individual events by default. Instead, the watch API tracks changes based on the backing datastore (PostgreSQL or CockroachDB). If a client lags behind, SpiceDB must retain these events in memory to ensure consistency guarantees.

Backpressure: Backpressure is handled at the transport level. If a consumer does not read data fast enough, the server-side write buffers fill up. This can lead to the connection being terminated by the OS or the SpiceDB process hitting an Out-of-Memory (OOM) limit.
  Stream Failure: The watch stream will fail if the consumer lags so far behind that the transaction token it is tracking is no longer available in the database's transaction log. In this case, the client must restart the watch from a snapshot.

Latency Scaling Factors
Latency scales with total write throughput rather than just writes matching subscribed object types. Because the watch mechanism must monitor the global transaction log to determine which events apply to which subscriptions, high-volume global writes increase the CPU overhead for the SpiceDB node to filter and dispatch events, even if a specific client isn't interested in those specific changes.

Recommended Metrics for Sharding
To determine if you need to shard watch consumers or optimize your architecture, monitor the following:
n
  spicedb_watch_queue_depth: This indicates how many events are waiting to be delivered to active streams.
  Process Memory Usage: Watch for linear growth during write bursts, which indicates that slow consumers are holding onto large event buffers.
  Event Latency: The time difference between the database commit timestamp and the time the client receives the event.

Note: This analysis assumes you are using a version of SpiceDB where the watch API relies on the database's change data-capture mechanism.

SpiceDB watch stream latency under high write throughput: what limits apply?

1 answer

Subscription Queueing and Backpressure

Latency Scaling Factors

Recommended Metrics for Sharding

Your perspective belongs here.

0 question comments