ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
DevOps & observability · #prometheus
The same project needs to behave consistently on developer machines, in CI and after deployment. Which versions, dependencies and configuration should be recorded?
Goal: Determine whether enabling the storage.tsdb.allow-overlap flag (introduced in Prometheus 2.30) affects the reliability of standard verification techniques used after restoring a TSDB snapshot. When a snapshot is restored, Prometheus replays the WAL to bring the database up to date, and verification typically relies on checking that known series contain
Goal I want Prometheus (assume a recent 2.x release) running inside a Kubernetes cluster to scrape only pods it can actually reach. My cluster uses NetworkPolicy to isolate several namespaces, and I want the target list to reflect that isolation without hand-maintained relabel rules for every policy change. Observed constraint From my reading of the kubernet
Evaluating Rancher memory leak detection options Goal: Choose between Rancher’s native /metrics endpoint and an external Prometheus stack from the monitoring catalog to identify upward memory trends in the server or agent components. Constraint: The Rancher HA setup recommends a metrics scrape interval of no more than 15 seconds to limit overhead, yet the ag
In current Prometheus releases, upgrades typically replace the binary and restart the process while the data directory remains unchanged, preserving existing time-series blocks. When the new binary aborts during startup because blocks fail checksum validation, operators face a choice between restoring a pre-upgrade backup of the data directory or accepting t
The goal is to determine whether Prometheus’ internal HTTP client connection pool is the root cause of intermittent connection errors observed during high‑frequency scrape or remote‑write operations. Because Prometheus reuses a single http.Client whose pool size is fixed by Go’s default transport and cannot be tuned via configuration, operators lack visibili
Prometheus enforces a global query concurrency limit via --query.max-concurrency, default 20. Excess queries queue and latency appears only when concurrent requests exceed the limit. The limit applies to user queries and internal workloads such as recording rule and alerting rule evaluations. Prometheus 3.0 introduced a new query engine with improved paralle
Prometheus remote_write accepts a remote endpoint URL and queue configuration, but as of version 2.54 the HTTP client used for write requests does not expose a configurable timeout. When a remote storage endpoint becomes unresponsive, the client may block indefinitely rather than aborting after a fixed duration. The retry logic for failed writes also appears
Query Timeout Behavior in Prometheus Prometheus utilizes a global --query.timeout flag to prevent resource exhaustion from long-running queries. While the HTTP API allows for per-request timeout overrides, the server must manage the lifecycle of the underlying PromQL engine process when these limits are reached or when a client abruptly closes the connection
rancher