Cutting Spicedb Policy Evaluation Latency with Built‑In Caching

Problem

In a distributed authorization system, each request to Spicedb can trigger a complex policy evaluation. When the same decision is requested repeatedly—such as a user checking access to a resource—Spicedb spends the same amount of compute time re‑evaluating the policy, even though the outcome is identical. This repeated work inflates latency and CPU usage, especially under high request volumes.

What Is Policy Caching?

Spicedb’s policy caching stores the result of a policy decision in memory for a short period. Subsequent identical requests hit the cache instead of re‑running the evaluation engine. The cache is keyed by the decision request (subject, action, resource, and optional context). When a policy is updated, Spicedb automatically invalidates any cached decisions that depend on the modified policy, ensuring that stale decisions are not served.

Key Configuration Options

• --enable-cache – Turns the cache on. It is disabled by default for safety.
• cache-size – Maximum memory (in megabytes) the cache can consume. Default is 256 MiB.
• cache-ttl – Time‑to‑live for each cached decision. Default is 5 minutes.
• cache-max-entries – Optional limit on the number of entries if you prefer a count‑based eviction policy.

Typical Usage Pattern

1. Start Spicedb with caching enabled.
2. Run your workload and record baseline latency.
3. Re‑run the workload with caching and compare metrics.
4. Monitor cache hit/miss rates and memory usage.

Configuring the Cache

Below is a minimal example of a Spicedb configuration file that enables caching. Replace the placeholders with your actual values.

# spicedb.yaml
server:
  enable_cache: true
  cache:
    size_mb: 512
    ttl_seconds: 300
    max_entries: 100000
# other server settings...

If you prefer the command‑line interface, the equivalent flags are:

spicedb serve \
  --enable-cache \
  --cache-size 512 \
  --cache-ttl 300 \
  --cache-max-entries 100000

Run the command as a user with sufficient privileges (typically root or the user that owns the Spicedb process). After starting, the cache will use up to 512 MiB of RAM. You can verify the allocation by checking the process memory via top or ps -o rss.

Validating Effectiveness

To confirm that caching is working, you need to measure latency and inspect cache metrics. The following steps outline a reproducible workflow.

1. Baseline Latency

   Run a representative workload without caching enabled. For example, use curl to request a decision 1,000 times:

   for i in $(seq 1 1000); do
     curl -s -o /dev/null -w "%{time_total}\n" \
       http://localhost:8443/api/v1/decision?subject=alice&resource=repo1&action=read
   done > baseline.txt
   

   Compute the average latency from baseline.txt.

2. Enable Caching

   Restart Spicedb with the cache flags as shown earlier.

3. Repeat the Workload

   Run the same curl loop and record the new latency distribution.

4. Check Metrics

   Spicedb exposes Prometheus‑style metrics at /metrics. Query the cache counters:

   curl http://localhost:8443/metrics | grep cache_hits
   curl http://localhost:8443/metrics | grep cache_misses
   

   A high cache_hits count relative to cache_misses indicates that the cache is being used effectively.

5. Policy Update Test

   Modify a policy that affects the decision you just tested (e.g., add or remove a rule). After the update, immediately query the same decision and observe that the cache entry is evicted. You can see the eviction in the metrics: cache_misses should rise for that request until the TTL expires.

6. Memory Check

   Use top -p $(pgrep spicedb) to confirm that memory usage does not spike beyond the configured cache-size value.

Trade‑offs and Limitations

• Stale Decisions – If policies change more frequently than the TTL, a cached decision might grant access that should be denied until the cache expires. Shorter TTLs reduce this risk but increase cache churn.
• Memory Footprint – The cache consumes RAM linearly with the number of stored decisions. On resource‑constrained hosts, a large cache can cause out‑of‑memory conditions or swap, degrading performance.
• Complex Policies – Policies that rely heavily on dynamic attributes or external data may not benefit as much, because each evaluation still requires fetching those attributes. Caching may still help, but the speedup is smaller.
• Invalidation Delays – In high‑concurrency environments, the automatic invalidation of cache entries on policy updates might lag, creating a short window where stale decisions are served.
• Hidden Bottlenecks – Turning on caching can mask issues in the policy store or network layer. If caching hides a slow database, you might not notice until the cache fills up or the TTL expires.

Actionable Next Steps

1. Start with a conservative cache size (e.g., 256 MiB) and TTL of 5 minutes. Monitor cache_hits and cache_misses using a Prometheus alerting rule.
2. Adjust the TTL based on your policy update cadence: shorter for highly dynamic policies, longer for stable ones.
3. Set up a CI test that runs the baseline and cached workloads, compares latency, and fails if the cache hit ratio drops below a threshold.
4. Enable memory alerts in your monitoring stack to avoid over‑commitment. If the Spicedb process exceeds the configured cache size by more than 10 %, investigate.
5. Document the cache configuration in your deployment playbook so that future engineers understand the trade‑offs and can tweak the parameters as the system evolves.

By following this approach, you can confidently reduce Spicedb’s decision latency, maintain consistency after policy changes, and keep an eye on the resource impact of the cache.