Which documents remain resident after Couchbase Server reaches its low-water mark?

Question

Which documents remain resident after Couchbase Server reaches its low-water mark?
The goal is to understand the residency behavior of documents in a Couchbase bucket when memory pressure triggers the managed cache ejection policy.
Couchbase Server keeps keys and metadata resident while values may be ejected based on an NRU policy once the bucket’s low‑water mark is exceeded. However, the server does not expose a per‑document guarantee about which values stay in RAM, and the effective working set is further reduced by metadata and service overhead, making the relationship between bucket quota and resident data non‑trivial.
Operators must decide whether to increase the RAM quota, reduce the working set, or accept higher read latency, but the decision is complicated by version‑dependent water‑mark defaults, varying overhead, and the lack of a deterministic residency rule.
Which document characteristics (size, access frequency, mutation pattern) affect the likelihood of staying resident after ejection?
Can operators predict the resident set using only bucket‑level counters and workload metrics?
Does raising the RAM quota proportionally increase the number of resident documents, or does the ejection policy shift the threshold in a non‑linear way?

Tasadduq Burney · Answer

What remains resident after the low-water mark

When a Couchbase bucket using value-only ejection crosses its low-water mark, the server keeps all document keys and their associated metadata resident in the managed cache. Only document values become eligible for eviction under the Not-Recently-Used (NRU) policy.

Confirmed facts vs. likely behavior

Confirmed: Keys, CAS values, flags, expiry timestamps, and vBucket mappings stay in RAM. Values are evicted based on NRU scores. There is no per-document guarantee that any specific value will remain resident.

Likely explanation: Frequently or recently accessed values tend to survive longer because their NRU scores are higher. Cold values are evicted first. Document size influences overall memory pressure but is not the primary eviction signal; a large, hot document may stay resident while a small, cold one is evicted.

Answering the operational questions

Document characteristics: Access recency and frequency dominate. Mutation patterns change metadata but do not directly protect a value from ejection.
Predicting the resident set: Bucket-level counters show trends (e.g., cache miss rate, resident item rate) but cannot predict the exact resident set. Uniform access patterns can make eviction appear nearly random.
Raising the RAM quota: Increasing the quota raises the threshold before ejection starts and can increase the number of resident values, but not linearly. Metadata overhead (roughly 64–96 bytes per item) and internal service overhead consume quota first, so the effective capacity for values is smaller than the raw quota suggests.

Verifying residency safely

To confirm behavior in your environment, inspect bucket statistics and sample keys without disrupting production:

Query the bucket stats endpoint and watch ep_mem_low_wat, ep_mem_high_wat, ep_cache_miss_rate, and ep_resident_items_rate:
```
curl -u <user>:<password> http://<node>:8091/pools/default/buckets/<bucketName>/stats
```
Use cbstats with the key and vbucket arguments to inspect per-item residency flags for a sample of keys after memory pressure has been induced.

Exact low-water mark thresholds and NRU scoring details vary between Couchbase Server versions (for example, 6.x versus 7.x), so treat version-specific numbers as assumptions until checked against your running server.

If you need exact threshold values or tuning advice, which Couchbase Server major version are you running?

Which documents remain resident after Couchbase Server reaches its low-water mark?