Gradual memory increase observed during continuous replication sessions in NimbleOS

Direct Answer

The replication buffer garbage-collection (GC) threshold in NimbleOS is a manual tunable that trades off peak memory against CPU overhead on the replication worker threads. For typical mixed-workload arrays, HPE field guidance historically starts at the factory default (commonly 70–80% of the replication buffer pool) and adjusts in 5–10% increments while monitoring buffer-memory delta, queue depth, and replication lag. There is no universal baseline because the default varies by NimbleOS release and array model, and some 6.x releases contain a known buffer-leak defect that makes any threshold appear ineffective until patched.

Considerations for Threshold Selection

• Replication concurrency and stream size – More concurrent sessions or larger average stream sizes increase buffer dwell time; a higher threshold may be needed to avoid constant reclamation churn.
• Network latency and jitter – Higher latency extends the time buffers sit in the queue, pushing peak memory upward between GC cycles.
• Array memory headroom – Dedupe, compression, snapshot metadata, and other services compete for the same DRAM pool. Leave enough margin so replication GC pauses do not trigger broader memory pressure.
• RPO sensitivity – Aggressive (low) thresholds increase CPU on replication workers and can add small, periodic latency spikes during reclamation, slightly extending RPO.
• Workload mix – If VRR, peer persistence, and backup replication run simultaneously, some OS versions support per-session-class thresholds; otherwise a single global threshold must accommodate the worst-case class.

Non-Disruptive Impact Measurement

1. On NimbleOS 6.5 or later, run show replication --buffer-stats every 15–30 minutes during a representative replication window. This isolates the replication buffer component from OS page cache, which show memory conflates.
2. Correlate buffer-used and buffer-reclaimed counters with show replication --detail queue-depth and replication-lag fields.
3. Plot the sawtooth pattern: peak memory just before reclamation vs. trough just after. The threshold should sit comfortably above the trough with headroom for the observed peak.
4. No CLI command forces an immediate GC cycle, so measurement is read-only and non-disruptive.

Baseline Guidance and Version Dependency

HPE field engineers typically recommend starting at the shipping default (often 70–80% of the replication buffer pool) for mixed workloads, then tuning in 5–10% steps. However, the exact default and whether a replication-service restart is required to apply a change both depend on the NimbleOS version and array model. Additionally, several 6.x releases (e.g., 6.2.x–6.4.x) exhibited a defect where replication completion callbacks failed to release buffers, causing linear, unbounded growth that no threshold could contain; this was fixed in later patches (6.5+ and 6.10.x).

One Missing Detail That Changes the Recommendation

What is the exact NimbleOS version and array model? With that information I can cite the documented default threshold, confirm whether a service restart is needed, and verify whether your build includes the buffer-leak fix or requires a hotfix before any tuning is meaningful.