Architecting Prometheus Remote Write for Long-Term Storage

The Local Storage Limitation

Prometheus is designed as a short-term monitoring system. By default, it stores time-series data on a local disk with a limited retention period. When you need to analyze trends over months or years, or query metrics across multiple geographically dispersed clusters, local storage becomes a bottleneck and a risk. The solution is Remote Write, a mechanism that offloads samples to a dedicated long-term storage backend.

The Smallest Suitable Design

For most environments, the simplest viable architecture involves a single Prometheus server acting as a scraper and a compatible remote storage backend (such as Thanos, Cortex, or VictoriaMetrics).

In this design, Prometheus continues to handle local alerting and short-term querying, but asynchronously pushes every ingested sample to the remote endpoint. The data is transmitted via HTTP POST using Snappy-compressed Protocol Buffers to minimize bandwidth consumption.

Configuration Example

To enable this, add a remote_write block to your prometheus.yml file. This configuration should be applied to the Prometheus server process with permissions to access the network endpoint of the backend.

remote_write:
  - url: "http://remote-storage-backend:9090/api/v1/write"
    # Optional: Use basic auth for secure boundaries
    basic_auth:
      username: "admin"
      password: "secret-password"

Trust and Data Boundaries

The trust boundary in a Remote Write setup sits between the Prometheus server (the producer) and the storage backend (the consumer).

• Producer Responsibility: Prometheus is responsible for the correct labeling and sampling of data. It does not perform deduplication before sending; it pushes everything it scrapes.
• Consumer Responsibility: The remote backend must validate incoming samples. Because the backend is often shared across multiple Prometheus instances, it must handle potential label collisions or overlapping time series to prevent data corruption.

Operational Health Checks

Remote Write is asynchronous. A successful scrape does not guarantee the data has reached the remote store. You must monitor the Prometheus server's own metrics to ensure the pipeline is healthy. Run these queries in the Prometheus UI to verify the state:

Metric	What it Indicates	Warning Sign
prometheus_remote_storage_samples_total	Total samples sent to remote storage.	Flatline or sudden drop in count.
prometheus_remote_storage_batches_failed_total	Number of failed HTTP batches.	Any non-zero increasing trend.
prometheus_remote_storage_samples_dropped_total	Samples dropped due to queue overflows.	Any value above zero indicates data loss.

Failure Modes and Risk Mitigation

The primary failure mode is a backend outage or network partition. When the remote endpoint is unreachable, Prometheus queues the data in a local write-ahead log (WAL) and memory.

Risks include:

• Memory Pressure: If the outage lasts long enough, the queue can consume significant system memory, potentially leading to Out-of-Memory (OOM) kills of the Prometheus process.
• Data Loss: Once the internal queue reaches its limit, Prometheus will drop the oldest samples to make room for new ones.
• Cardinality Explosion: If a service suddenly generates millions of unique label combinations (high cardinality), the volume of Remote Write traffic can saturate the network interface or crash the backend.

When to Pivot the Design

The single-server push model fails when the volume of metrics exceeds the throughput capacity of a single Prometheus instance's remote writer. You should move to a distributed architecture (e.g., adding a load balancer or a dedicated distributor layer) if you observe the following:

1. Consistent Queue Backlog: The prometheus_remote_storage_batches_failed_total is low, but the samples are not arriving at the backend in real-time.
2. CPU Saturation: The Prometheus server spends a disproportionate amount of CPU on Snappy compression for remote writes.
3. Backend Bottlenecks: The remote storage backend reports 429 (Too Many Requests) or 503 (Service Unavailable) errors during peak traffic.

Rollback Procedure

If Remote Write causes instability (e.g., OOM kills), remove the remote_write block from prometheus.yml and reload the configuration using curl -X POST http://localhost:9090/-/reload. This immediately stops the outbound data stream and clears the memory queue.