Streamlining Long‑Term Metrics with Prometheus Remote Write

Why Remote Write Matters

Prometheus is great for short‑term, high‑resolution metrics, but its local storage is limited to a few days. If you need to keep data for months or years, you need a long‑term backend. remote_write lets a Prometheus instance stream samples to an external system (Thanos, Cortex, InfluxDB, or any HTTP‑based store) while keeping the local instance lightweight.

Core Concepts

• Asynchronous streaming: Prometheus buffers samples in memory and retries on failure.
• One‑way flow: data never comes back from the remote store to Prometheus.
• Configurable per‑job or global: you can target multiple backends or just one.

Configuring remote_write

The configuration is a single block under global: or inside a job definition. Below is a minimal example that writes to a Thanos Store Gateway over TLS with basic auth.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  remote_write:
    - url: "https://store-gateway.example.com/api/v1/receive"
      basic_auth:
        username: "prom_user"
        password: "$PROM_USER_PASSWORD"
      tls_config:
        ca_file: "/etc/prometheus/ca.crt"
        cert_file: "/etc/prometheus/client.crt"
        key_file: "/etc/prometheus/client.key"
      write_relabel_configs:
        - source_labels: ["__name__"]
          regex: "^node_.*"
          action: keep

Key points:

• url is the HTTP endpoint of the remote store.
• Secrets should be stored in files or a secrets manager; avoid hard‑coding passwords.
• The write_relabel_configs block lets you filter or transform metrics before they leave Prometheus.
• For a multi‑tenant setup, add another remote_write entry with a different URL or authentication.

Verifying the Setup

1. Check connectivity
   Run from the Prometheus host:

   curl -I https://store-gateway.example.com/api/v1/receive
   

   The response should be HTTP/2 200 and TLS handshake must succeed.
2. Inspect Prometheus metrics
   Query prometheus_remote_storage_write_total and prometheus_remote_storage_write_failures_total on the /metrics endpoint. During a test scrape they should increase.
   • Example: curl http://localhost:9090/metrics | grep prometheus_remote_storage_write_total
3. Monitor the remote backend
   Check its ingestion rate and latency dashboards. They should be within acceptable limits for your scrape interval.
4. Look at logs
   Prometheus logs contain entries like remote write error when a write fails. Verify that retries happen and that failures are logged only when the buffer is full.

Trade‑offs and Limitations

Aspect	Benefit	Risk / Cost
Network Usage	Minimal local load	Higher outbound traffic; monitor bandwidth.
Data Retention	Months/years of metrics	Remote store storage costs.
Buffer Size	Graceful handling of transient outages	Large buffers consume memory; configure remote_write.buffer_size appropriately.
Metadata Preservation	All labels sent	Some backends drop annotations or non‑standard labels.

Because remote_write is one‑way, you cannot query the remote store directly from Prometheus. For full query support you need a companion system (e.g., Thanos Query or Cortex Query) that aggregates the remote data.

Actionable Checklist

• Define the remote store URL and secure credentials.
• Add remote_write to the Prometheus config, test connectivity.
• Enable write_relabel_configs to drop noisy metrics if needed.
• Verify prometheus_remote_storage_write_total increments during a test scrape.
• Set remote_write.buffer_size to at least twice your peak write rate.
• Monitor failure metrics and buffer length; adjust buffer or backend capacity as needed.
• Document the configuration and alerting rules for remote write failures.

With these steps you can confidently offload long‑term storage from Prometheus while keeping the local instance responsive for real‑time dashboards.