Diagnosing and Fixing Memory‑Optimized Bucket Quota Exhaustion in Couchbase 7.x

Recognizable Condition

Clients report intermittent write latency spikes, occasional timeout or out_of_memory errors, and a sudden rise in operation timeouts in application logs. Cluster‑wide metrics show the active eviction rate on data nodes rising above 0 % and the memory used approaching the configured memory quota for the bucket. CPU on the data service climbs during eviction periods.

Cause Diagnostic Table

Potential Cause	What to Look For
Memory quota too small for working set	Memory used ≈ quota; active eviction > 0 % for > 5 min
Large value items or growing documents	Average item size > 200 KB; size distribution skewed to > 1 MB
Write‑heavy workload without TTL	Item count steadily increasing; no expiration set
Replica count higher than needed	Replica count per bucket > 1; each replica consumes full memory copy
Compaction or index pressure	Index memory usage high; data service memory drops after compaction

Ordered Checks

1. Verify per‑node memory usage. Run on a data node:

   # cbstats -u admin -p password -c node_ip | grep "data memory used"
   # cbstats -u admin -p password -c node_ip | grep "data memory quota"
   

   Check that data memory used is < 90 % of data memory quota under steady load.
2. Inspect eviction metrics. On the cluster console or via REST:

   curl -s -u admin:password http://node_ip:8091/pools/default/buckets/bucket_name | jq .evictionPolicy
   curl -s -u admin:password http://node_ip:8091/pools/default/buckets/bucket_name | jq .evictionRate
   

   A non‑zero evictionRate indicates active eviction.
3. Analyze document size distribution. Use the SDK or N1QL:

   SELECT AVG(JSON_LENGTH(value)) AS avg_size, COUNT(*) AS count
   FROM `bucket_name`
   GROUP BY bucket_name
   LIMIT 1;
   

4. Check write throughput and TTL usage. Run a simple write‑rate test or review application logs for set operations without expiry.
5. Confirm replica configuration. In the bucket settings or via REST:

   curl -s -u admin:password http://node_ip:8091/pools/default/buckets/bucket_name | jq .replicaNumber
   

6. Review index and compaction impact. Monitor index memory usage and compaction events in cluster logs.

Fixes Tied to Findings

• Increase memory quota or add data nodes. Adjust memory quota in bucket settings or scale the cluster, then run rebalance.
• Reduce large document sizes. Refactor payloads, enable compression, or move large blobs to a separate bucket or external storage.
• Introduce TTL or cleanup old data. Add expiry on new writes or run a one‑time cleanup script to delete stale items.
• Lower replica count. If durability requirements allow, reduce replicaNumber to 1 or 0, then rebalance.
• Adjust eviction policy. For memory‑optimized buckets, value_only is the default; consider full_eviction if the bucket holds very large values that can be evicted.

Escalation Criteria

• Eviction rate remains > 5 % after increasing quota and reducing data volume for > 15 min.
• Client errors persist with disk_full or out_of_memory after all memory‑level fixes.
• Rebalance fails or nodes become unhealthy during quota changes.
• Durability constraints cannot be met due to reduced replicas, indicating a deeper architectural issue.

Practical Example

In a 3‑node cluster, bucket app_data is memory‑optimized with a 512 MB quota. After a spike in writes, the admin sees data memory used 490 MB and eviction rate 12 %. The following steps were taken:

1. Run cbstats on each node; confirm all nodes at 95 % usage.
2. Query average document size: SELECT AVG(JSON_LENGTH(value)) AS avg_size FROM `app_data` LIMIT 1; returns 350 KB.
3. Reduce average size by splitting large JSON objects into two smaller documents and enabling compression.
4. Add expiry of 30 days to new writes.
5. Increase quota to 640 MB via the web console and trigger rebalance.
6. After 30 min, data memory used drops to 420 MB and eviction rate is 0 %.
7. Monitor client latency: 95th percentile drops from 600 ms to < 50 ms.

Limitations

• Metrics names and units can vary between 7.x releases; verify against your cluster’s pools API.
• Rebalancing can temporarily degrade performance; schedule during maintenance windows.
• Disk‑optimized buckets behave differently; this guide applies to memory‑optimized buckets only.

Verification Checklist

1. Cluster reports data memory used < 90 % of quota under steady load.
2. Active eviction rate is 0 % in the last 10 minutes.
3. Client write latency percentiles return to baseline (< 50 ms for 95th percentile).
4. Bucket item count and average size trend stabilizes or decreases.