Offloading Long‑Term Metrics with Prometheus Remote Write: A Practical Guide

Problem: Prometheus’ Built‑In Retention is Too Short

Prometheus keeps data only for the period you configure (default 15 days). In a busy cluster you quickly outgrow that window, but you still need to keep the history for compliance, trend analysis, and cross‑service correlation. Simply increasing the local retention is not a scalable solution – disk grows linearly and query performance drops.

Thesis: Remote Write Lets You Keep the Best of Both Worlds

Prometheus’ remote_write block streams every sample to a downstream storage that can keep the data indefinitely. The local Prometheus instance continues to serve recent data with low latency, while the remote system handles long‑term retention, compression, and global federation.

How Remote Write Works

The remote_write section in prometheus.yml instructs the TSDB to forward samples as they are ingested. The downstream endpoint receives the data in the Prometheus Remote Write protocol (a series of WriteRequest protobuf messages). Most long‑term backends – Thanos Store, Cortex, VictoriaMetrics – expose an HTTP endpoint that accepts these writes.

Key configuration items:

• url – the HTTPS endpoint of the remote store.
• remote_timeout – how long to wait for a successful write before retrying.
• basic_auth – optional credentials if the endpoint is protected.
• headers – custom headers (e.g., Authorization: Bearer …) for token‑based auth.

Worked Example: Prometheus → Thanos Store

Assume you already have a Thanos Store Gateway running in the same Kubernetes namespace. The endpoint is https://thanos-store.example.com/api/v1/receive and requires a bearer token.

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "kubernetes-nodes"
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        target_label: __address__
        replacement: ${HOST_IP}:9100

remote_write:
  - url: https://thanos-store.example.com/api/v1/receive
    remote_timeout: 10s
    basic_auth:
      username: "remote"
      password: "${THANOS_TOKEN}"
    headers:
      X-Forwarded-For: "${HOST_IP}"

Steps to validate:

1. Check syntax – run promtool check config prometheus.yml to catch YAML errors.
2. Restart Prometheus – the sidecar will start streaming.
3. Verify ingestion – use curl -k -H "Authorization: Bearer ${THANOS_TOKEN}" https://thanos-store.example.com/api/v1/query?query=up and look for the current up{instance="…"} sample.
4. Inspect latency – in Grafana, add a panel that queries remote_write_latency_seconds{job="prometheus"} to see the round‑trip time.

Trade‑Offs and Limitations

• Increased Latency – every sample is sent over the network, adding a few seconds of delay before it appears in the remote store. This is acceptable for historical queries but not for real‑time alerts.
• Network and CPU Load – high scrape rates can saturate the network or overwhelm the remote endpoint, leading to dropped samples. Thanos Store can be tuned with receive.max_outstanding_requests to throttle backpressure.
• Duplication Risk – if multiple Prometheus instances scrape the same target and all are configured to write to the same store, you’ll get duplicate series. Use a deduplication_key or ensure only one instance is responsible for the scrape.
• Local Storage Still Needed – Remote Write does not replace the local TSDB. You must keep a retention window (e.g., 15 days) for quick queries and alerts.

Practical Takeaway

Remote Write is a lightweight, declarative way to offload long‑term storage without changing your existing scrape configuration. Deploy a Thanos or Cortex sidecar, add the remote_write block, and you’ll automatically get compressed, globally federated metrics that survive months or years. Just remember to monitor write latency, network capacity, and potential duplication when scaling out.