Offloading Prometheus Metrics to Long‑Term Storage with Remote Write

Problem: Storage Bloat and Slow Queries on a Single Prometheus

Prometheus is built for short‑term, high‑cardinality metrics. A single instance keeps all samples in a local TSDB, so retention is limited (default 15 days). When you try to keep 30–90 days of data or run a cluster with many targets, the disk grows fast, and query latency spikes as the TSDB scans more blocks.

Typical symptoms:

• Disk usage exceeds 80 % within weeks.
• Querying a 24‑hour window takes >5 s.
• Prometheus restarts due to insufficient space.

Thesis: Decouple Storage from Query with Remote Write

Prometheus can push every sample it receives to an external system via the remote_write API. The remote backend stores the samples, while the local TSDB can be configured for a short retention (e.g., 1 day). Queries are served from the long‑term store (Thanos Querier, Cortex Querier, or InfluxDB), keeping the local instance lightweight and queries fast.

Remote‑Write Architecture

The flow is:

• Prometheus receives metrics from scrape targets.
• For each sample, it writes locally and enqueues a remote_write request.
• The request is sent over HTTP/HTTPS to the remote endpoint. The body is a WriteRequest protobuf (RFC 5325).
• Remote storage accepts the protobuf, persists it, and optionally indexes for fast queries.
• When a user queries, the Querier fetches data from the remote store; the local TSDB is only used for the short window.

Key requirements for the backend:

• HTTP/HTTPS support with optional TLS.
• Authentication (Basic, Bearer, or client cert).
• Accepts the protobuf format and returns 200 OK.
• Can ingest data at the rate Prometheus scrapes.

Configuration Example

Below is a minimal prometheus.yml that pushes to a Thanos Store Gateway. Replace placeholders with your values.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: "prod"

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

remote_write:
  - url: "https://thanos-store-gateway.example.com/api/v1/receive"
    remote_timeout: 30s
    queue_config:
      max_samples_per_send: 5000
      max_shards: 200
      queue_size: 100000
    tls_config:
      ca_file: "/etc/prometheus/certs/ca.crt"
      cert_file: "/etc/prometheus/certs/client.crt"
      key_file: "/etc/prometheus/certs/client.key"
    basic_auth:
      username: "promuser"
      password: "prompass"
    headers:
      X-Forwarded-For: "{{ .RemoteAddr }}"
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "^node_cpu_seconds_total$"
        action: keep

Explanation of key fields:

• remote_timeout – maximum time to wait for the remote store to acknowledge a batch.
• queue_config – controls how many samples are buffered before sending. Adjust for high‑cardinality workloads.
• tls_config – ensures encrypted traffic and mutual TLS if required.
• basic_auth – optional; use if the remote store expects HTTP Basic auth.
• write_relabel_configs – optional filtering; useful to drop noisy metrics before sending.

Worked Example: 2‑Node Prometheus Cluster to Thanos Store Gateway

Assume you have two Prometheus instances behind a load balancer, each configured with the above prometheus.yml. Steps to validate:

1. Verify Prometheus supports remote write: prometheus --version should output a version ≥ 2.15.
2. Reload configuration: curl -X POST http://localhost:9090/-/reload. Check the response for success.
3. Confirm the remote block is active: curl http://localhost:9090/api/v1/status/config | jq '.data.remote_write[0].url' should return the Thanos endpoint.
4. Send a test metric: Use curl -X POST -H "Content-Type: application/x-protobuf" --data-binary @test.pb https://thanos-store-gateway.example.com/api/v1/receive. Replace test.pb with a small protobuf payload you crafted.
5. Query the remote store: Open Grafana, add a Thanos Querier data source pointing to the same gateway, and query the metric you sent. If it appears, the pipeline works.
6. Optional: run promtool check config prometheus.yml to catch syntax errors before reload.

After confirming, monitor the prometheus_remote_write_samples_sent_total counter to ensure data is flowing and the prometheus_remote_write_error_total stays at zero.

Trade‑Offs and Limitations

• Network overhead: Each sample is duplicated over the network. In high‑cardinality scenarios, the bandwidth cost can be significant. Use max_samples_per_send and queue_size to balance latency vs. throughput.
• Data duplication: If you keep a long retention in the local TSDB and also in the remote store, you’ll pay for storage twice. Align retention in prometheus.yml with the remote backend’s retention policy.
• Authentication complexity: Mis‑configured TLS or auth can silently drop samples. Verify with promtool check config and check the remote logs for 401/403 responses.
• Query path: Queries to the remote store may be slower than local ones, especially for very short windows. Thanos uses a hybrid approach: it serves 1‑day queries from the local TSDB and longer queries from the store.
• Recovery: If the remote store goes down, Prometheus will buffer samples until the queue is full. Configure max_shards and queue_size to avoid data loss.

Actionable Checklist

• Confirm Prometheus version ≥ 2.15.
• Set retention to a short period (e.g., 24 h) in prometheus.yml.
• Configure remote_write with TLS and auth matching your backend.
• Validate config with promtool check config and reload.
• Verify the remote block is active via /api/v1/status/config.
• Send a test metric and confirm it appears in the remote store.
• Monitor prometheus_remote_write_samples_sent_total and error counters.
• Adjust queue_config if you see backpressure or high latency.
• Align local retention with remote retention to avoid duplicate storage.

By offloading metrics with remote write, you keep Prometheus fast and lightweight while still enjoying long‑term retention and scalable query performance. The key is a careful configuration that balances network usage, authentication, and retention alignment.