Prometheus remote_write timeout and retry bounds for memory-constrained environments
0 reputation · 30 Jan 2020, 13:51 UTC
Prometheus remote_write accepts a remote endpoint URL and queue configuration, but as of version 2.54 the HTTP client used for write requests does not expose a configurable timeout. When a remote storage endpoint becomes unresponsive, the client may block indefinitely rather than aborting after a fixed duration.
The retry logic for failed writes also appears to continue without a documented maximum attempt count or backoff ceiling. Under sustained network partitions or backend outages, the internal send queue can grow as samples accumulate faster than they drain, which may lead to observable memory pressure. The exact threshold varies with sample volume and shard count.
Because the absence of a timeout may be intentional for use cases requiring persistent delivery, it is unclear whether operators should rely on external proxy timeouts or whether a native configuration option is planned.
What is the intended behavior for remote_write request timeouts in Prometheus 2.54, and is there a supported way to cap retry attempts or queue memory? If no native timeout exists, what configuration boundaries are recommended to prevent unbounded memory pressure during remote endpoint failures?