Designing Prometheus Remote Write: Requirements, Minimal Architecture, and Operational Safeguards

Problem Statement

Prometheus needs to ship ingested samples to a remote storage system reliably, without losing data during transient network issues, while preserving TLS authentication and providing back‑pressure awareness. The goal is to define the smallest architecture that satisfies these requirements and to outline the operational checks and failure modes that inform when the design must evolve.

Requirements

• Reliable ingestion of samples from Prometheus to remote storage.
• Support for back‑pressure so the sender can slow down when the receiver is overwhelmed.
• TLS authentication (bearer token or mTLS) and certificate validation.
• At‑least‑once delivery semantics; exact ordering is not required.
• Low operational overhead: metrics to monitor queue length, dropped samples, and batch latency.

Smallest Suitable Design

The minimal implementation reuses Prometheus’ built‑in remote write client: a sidecar‑like thread inside the Prometheus server that reads the write‑ahead log (WAL), batches samples, compresses them with Snappy, and sends them over HTTP/2 to the configured endpoint. No separate process is required; the existing remote_write configuration drives the behavior.

Configuration Example

remote_write:
  - url: "https://remote-storage.example.com/api/v1/write"
    basic_auth:
      username: "prometheus"
      password: "${REMOTE_WRITE_PASSWORD}"
    tls_config:
      ca_file: /etc/prometheus/ca.crt
      cert_file: /etc/prometheus/client.crt
      key_file: /etc/prometheus/client.key
    queue_config:
      max_samples_per_send: 5000
      max_shards: 200
      capacity: 25000
      min_backoff: 10s
      max_backoff: 300s
      min_retry: 10s
      max_retry: 300s

Placeholders like ${REMOTE_WRITE_PASSWORD} should be replaced with a secret‑management mechanism appropriate for your environment (e.g., Kubernetes secrets, Vault, or envsubst).

Trust and Data Boundaries

The remote endpoint is treated as untrusted. Prometheus authenticates via bearer token or mTLS and validates the server’s TLS certificate chain. Data is protected in transit (TLS + Snappy compression) but no assumption is made about encryption at rest on the remote side; that responsibility lies with the storage system.

Operational Checks

Key metrics to monitor (all prefixed with prometheus_remote_storage_):

• queued_samples – number of samples waiting to be sent; sustained growth indicates back‑pressure or receiver unavailability.
• failed_samples_total – cumulative count of samples dropped due to permanent failures (e.g., authentication errors, unrecoverable network errors).
• send_batch_duration_seconds – latency of each batch; rising latency can hint at network congestion or receiver slowness.

Set alerts on:

1. Queued samples > 80 % of queue_config.capacity for >5 minutes.
2. Failed samples rate > 0.1 % of incoming samples over 1 minute.
3. Batch latency > 2 × baseline for >3 minutes.

Failure Modes and Design Triggers

Unreachable Endpoint

If the remote endpoint is down, samples accumulate in the WAL. When the WAL reaches storage.tsdb.max‑bytes (default 50 % of disk), Prometheus begins dropping samples, incrementing failed_samples_total. This is a hard failure mode that requires operator intervention.

Frequent Network Partitions

Repeated short outages cause the queue to oscillate, increasing CPU usage due to retries. If observed, consider:

• Enabling a larger queue_config.capacity to absorb bursts.
• Increasing WAL size (storage.tsdb.max‑bytes) to tolerate longer outages.
• Moving to a federated or Thanos‑based architecture where remote write is replaced by object storage.

Version Incompatibility

Prometheus ≤ 2.20 lacks certain flow‑control improvements in the remote write client. If the remote storage expects a newer protocol version (e.g., handling of X-Prometheus-Remote-Write-Version header), silent sample loss can occur. Verify both sides advertise the same version; upgrade Prometheus if needed.

When to Reconsider the Design

Redesign is warranted when:

• Operational checks show sustained queue growth despite increased capacity and back‑off tuning.
• Disk usage approaches the WAL limit regularly, risking sample loss.
• The remote storage cannot keep up with peak ingestion rates, causing chronic back‑pressure.
• Operational overhead (CPU/I/O) from remote write exceeds acceptable limits for the host.

In those cases evaluate alternatives such as:

• Thanos Receive or Cortex for scalable remote write.
• Agent‑based solutions (e.g., Prometheus Agent) that decouple ingestion from storage.
• Adjusting scrape intervals or reducing label cardinality to lower sample volume.

Verification Steps (No Claim of Testing)

To validate the design in a staging environment:

1. Deploy a minimal mock endpoint (e.g., a Python Flask app) that logs the received Snappy‑compressed batches and returns 200.
2. Point Prometheus to the mock using the configuration above.
3. Under normal load, confirm that prometheus_remote_storage_queued_samples stays near zero and failed_samples_total does not increase.
4. Stop the mock endpoint; observe the queue metric rise. Allow the WAL to fill until storage.tsdb.max‑bytes is reached, then verify that failed_samples_total increments.
5. Restart the mock endpoint; check that the queue drains, latency returns to baseline, and no additional samples are lost beyond the expected dropped count.

These steps provide a practical way to check that the remote write behaves as designed without asserting production‑grade validation.

Limitations

• Remote write adds CPU and I/O overhead for WAL reading, batching, compression, and HTTP/2 framing; ensure the host has sufficient headroom, especially under high churn scenarios.
• Exact ordering of samples is not guaranteed; downstream systems must tolerate out‑of‑order samples.
• The design assumes the remote storage can handle HTTP/2 and Snappy; if not, fallback to HTTP/1.1 may be required, affecting performance.