Tuning Prometheus Remote Write Queues to Stop Sample Loss in Growing Clusters

A growing Kubernetes cluster pushes more scrape targets into Prometheus. When the remote write endpoint slows, the local write-ahead log fills, the in-memory queue backs up, and samples get dropped. The result is gaps in long-term storage and alerting blind spots exactly when traffic is highest.

The useful takeaway is that sample loss is usually backpressure, not a bug. Sizing the remote write queue, shards, and retry backoff lets Prometheus absorb bursts without unbounded memory growth.

How remote write moves samples out

Prometheus writes scraped samples to the WAL first. The WAL is an append-only log on disk that survives restarts. A background send loop reads batches from the WAL and pushes them to remote storage via remote_write.

Each remote_write target has a queue. The queue holds samples waiting to be sent. Shards are independent workers that pull from the queue. If the remote endpoint is slow or returns 5xx, the send loop backs off and retries with exponential delay. When the queue is full, new samples are dropped and exposed by prometheus_remote_storage_samples_dropped_total.

Tuning queue_config for predictable backpressure

Queue tuning controls three things: concurrency, batch size, and retry behavior.

• max_shards: number of parallel send workers per target. More shards increase concurrency to the remote endpoint.
• max_samples_per_send: batch size per request. Larger batches reduce request overhead but increase latency per batch.
• min_backoff and max_backoff: retry delay bounds when the remote endpoint is unavailable.

These fields are available in Prometheus v2.30 and later. Validate any change before reload.

# run on the Prometheus host or CI runner with read access to the config
promtool check config /etc/prometheus/prometheus.yml

Expected check: no errors printed and exit code 0. Risk: a bad config will cause the reload to be rejected and the running config stays unchanged. Test in staging first.

Worked example: defaults vs tuned

Default queue_config is conservative: 1 shard, 1000 samples per send, 5s to 100s backoff. For a busy cluster this can under-utilize the remote endpoint.

Example tuned block for a single remote_write target:

remote_write:
  - url: https://remote.example.com/api/v1/write
    queue_config:
      max_shards: 5
      max_samples_per_send: 10000
      batch_send_deadline: 5s
      min_backoff: 250ms
      max_backoff: 10s

Placeholders: replace the URL with your remote storage endpoint. The config is applied by reloading Prometheus, e.g. via SIGHUP or the API reload endpoint. This changes runtime behavior; keep a rollback copy of the previous config.

Trade-off: memory and visibility lag

Larger queues and more shards absorb spikes, but they increase memory use in the Prometheus process. A very large queue can delay the point at which backpressure is felt, which means recent data may sit in memory longer before it is visible in remote storage.

Extreme queue sizes can exhaust container memory and trigger OOM kills. Size conservatively and watch process RSS.

Operational checklist

• Monitor prometheus_remote_storage_queue_length per target. Sustained growth indicates the sender cannot keep up.
• Alert on increase of prometheus_remote_storage_failed_samples_total and prometheus_remote_storage_samples_dropped_total.
• After a config change, inspect queue length and failed samples to confirm the trend moves toward zero growth.
• Review queue_config quarterly or after a significant increase in scrape targets.

Reloads are safe when config passes promtool check config, but avoid changing queue size during known peak load. Keep changes small and observable.