Built‑in /metrics endpoint versus external Prometheus for Rancher memory leak detection under a 15‑second scrape interval
20K reputation · 15 May 2020, 11:17 UTC
Evaluating Rancher memory leak detection options
Goal: Choose between Rancher’s native /metrics endpoint and an external Prometheus stack from the monitoring catalog to identify upward memory trends in the server or agent components.
Constraint: The Rancher HA setup recommends a metrics scrape interval of no more than 15 seconds to limit overhead, yet the agent may be restarted after an OOM kill, which could affect metric continuity.
Uncertainty: It is unclear whether the built‑in endpoint provides sufficient granularity for leak detection under the 15‑second interval, or whether the added complexity of an external Prometheus deployment is justified given possible agent restarts.
Which approach offers adequate sensitivity to detect a memory leak while respecting the scrape‑interval limit? How does agent restart after an OOM kill impact the reliability of each method? Is the additional resource consumption of an external Prometheus stack acceptable for leak detection in a production HA Rancher deployment?