Configuring Prometheus Remote Write: Queue Tuning, TLS, and the Mistakes That Drop Your Samples

Remote write is how Prometheus forwards every ingested sample to an external store — Cortex, Mimir, Thanos Receive, InfluxDB, or any endpoint speaking the remote write protocol — while keeping its local TSDB for recent queries. The useful answer up front: a working setup needs only a remote_write block with a URL, but a reliable setup needs deliberate queue_config tuning and monitoring of the prometheus_remote_storage_* metrics, because when the in‑memory queue fills, Prometheus silently drops samples and only logs a warning.

How remote write actually works

Prometheus scrapes targets and writes samples to its local TSDB as usual. In parallel, the remote storage subsystem tails the TSDB’s write‑ahead log (WAL) and re‑ships those samples to each configured remote endpoint. Because it reads from the WAL rather than scraping twice, remote write adds modest overhead and survives restarts — samples already in the WAL are retried after a crash.

Each endpoint gets its own sharded queue. Shards are worker goroutines that batch samples into snappy‑compressed protobuf WriteRequest messages and POST them. The queue scales shard count dynamically between min_shards and max_shards based on pending backlog. If the remote endpoint or network can’t keep up and every shard’s buffer (capacity samples each) fills, new samples are dropped.

A worked configuration

The following prometheus.yml fragment (valid for Prometheus 2.x; run promtool check config prometheus.yml before reloading) sends everything to a TLS‑protected endpoint with conservative queue settings:

remote_write:
  - url: https://remote-storage.example.com/api/v1/write
    send_exemplars: true
    queue_config:
      capacity: 2500
      max_shards: 200
      min_shards: 1
      max_samples_per_send: 500
      min_backoff: 30s
      max_backoff: 300s
    tls_config:
      ca_file: /etc/prometheus/ca.pem
      cert_file: /etc/prometheus/client.pem
      key_file: /etc/prometheus/client-key.pem

What the key knobs mean:

• capacity — samples buffered per shard before dropping. Total worst‑case buffer is roughly capacity × max_shards, which directly drives memory use.
• max_samples_per_send — batch size per POST. Larger batches reduce request overhead but increase the cost of a single failed send.
• min_backoff/max_backoff — retry delay on failed sends, with exponential backoff between retries. Failed batches are retried, not dropped immediately.
• send_exemplars — forward exemplars (trace‑linked sample annotations) if your backend supports them.

Apply changes with a config reload (kill -HUP <pid> or POST to /-/reload if --web.enable-lifecycle is set). Note that changing queue_config historically required a restart to take full effect on some versions — check your version’s changelog if a reload seems ignored.

Verifying it works

Do not assume success from the absence of errors at startup. Check three things:

1. Config validity: promtool check config prometheus.yml catches YAML and schema errors before reload.
2. Logs: look for remote storage startup messages and, critically, the absence of repeated "failed to send batch" warnings.
3. Metrics: query Prometheus itself for prometheus_remote_storage_samples_dropped_total (should stay at zero), prometheus_remote_storage_samples_failed_total, prometheus_remote_storage_queue_highest_sent_timestamp_seconds versus prometheus_remote_storage_wal_watcher_current_segment, and shard utilization via prometheus_remote_storage_shards compared to prometheus_remote_storage_shards_desired. If desired shards are pinned at max and the pending‑samples metric grows, the endpoint can’t keep up.

For an end‑to‑end test, point the URL at a throwaway receiver — a small HTTP service that logs POST bodies, or a second Prometheus started with --web.enable-remote-write-receiver — and confirm samples arrive and are queryable there.

Limits you need to design around

Remote write throughput is bounded by network bandwidth and the remote endpoint’s ingestion rate, not by Prometheus’s scrape capacity. A Prometheus happily ingesting 500k samples/second locally can still fall behind on a slow or rate‑limited endpoint, and the queue only buys time — it is a buffer, not storage.

Memory is the other constraint. With capacity: 2500 and max_shards: 200, the worst‑case buffer is 500,000 samples per endpoint. Multiply by the number of remote_write entries and size your container or host accordingly; oversized queues are a common cause of OOM kills during endpoint outages.

Finally, remote write is not an acknowledgment‑based replication protocol. Prometheus retries failed batches with backoff, but samples dropped from a full queue are gone. If you need guaranteed durability, that guarantee must come from the backend architecture (e.g., replicated receivers), not from remote write itself.

Common mistakes

• Wrong URL path. Most backends expect /api/v1/write or /api/v1/push; a bare hostname returns 404s that surface only as repeated send failures.
• Missing TLS or auth config. Connection failures from certificate or credential problems look identical to endpoint downtime in the logs — test the URL with curl from the Prometheus host first.
• Queue tuned too small. Low capacity or max_shards causes drops during brief endpoint hiccups that a larger buffer would have absorbed.
• Protocol version mismatch. Confirm the backend implements remote write protocol 0.1.0 or later; mismatches produce protobuf parse errors on the receiver side.
• No alerting on drop metrics. Dropped samples are silent data loss. Alert on any increase in prometheus_remote_storage_samples_dropped_total.

The practical check that ties it together: after any change, watch the dropped‑samples counter and the highest‑sent‑timestamp lag for at least one full scrape‑heavy period. Zero drops and a lag near zero mean the pipeline is healthy; anything else means the queue, network, or backend needs attention before you trust the long‑term data.