Prometheus Storage: Choosing Between Local TSDB and Remote Write

The Storage Capacity Dilemma

Prometheus is designed as a short-term monitoring system. By default, it stores data in a local Time Series Database (TSDB), which is highly efficient for immediate alerting and operational dashboards. However, as your infrastructure grows, you will hit a wall: either your disk fills up, or your query performance drops because you are scanning too many historical blocks.

The core decision is whether to rely on Local TSDB storage for all needs or to implement Remote Write to offload data to a long-term storage backend. The goal is to balance the need for high-resolution historical data against the resource constraints of the Prometheus server.

Comparing Storage Strategies

Feature	Local TSDB Storage	Remote Write (External Backend)
Retention	Limited by disk space	Virtually unlimited (S3/GCS/etc.)
Query Speed	Fast for recent data	Variable; depends on backend
Availability	Single point of failure	High (via distributed backends)
Resource Cost	Disk I/O and space	CPU and Network bandwidth
Complexity	Zero configuration	Requires managing a second system

Trade-offs and Engineering Constraints

Local TSDB: The Operational Path

Local storage uses a block-based structure where samples are written to memory and periodically flushed to disk. A background process called compaction merges these blocks to keep queries efficient. This is ideal for "operational" monitoring—where you only care about the last 15 to 30 days of data to trigger alerts or debug a current incident.

The primary risk is disk I/O saturation. If your churn rate (the number of new time series created per hour) is high, compaction can consume significant disk bandwidth, potentially slowing down ingestion.

Remote Write: The Analytical Path

Remote Write allows Prometheus to stream samples to an external system (such as Thanos, Cortex, or VictoriaMetrics) via HTTP. This decouples ingestion from retention. You can keep only 2 hours of data locally for alerting while storing 2 years of data remotely for capacity planning.

The trade-off is an increase in CPU and network utilization. Because Prometheus must compress and transmit every sample, a high-cardinality environment can see a noticeable spike in CPU usage on the Prometheus pod/server.

Implementation and Validation

Scenario A: Extending Local Retention

If you have sufficient disk space and do not need a global view across multiple clusters, you can extend the local retention period. This is configured via a command-line flag during the Prometheus startup process.

# Run Prometheus with a 30-day retention period
./prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=30d

Risk: Setting this too high without monitoring disk usage can lead to No space left on device errors, which will crash the Prometheus process.

Scenario B: Configuring Remote Write

To offload data, add a remote_write block to your prometheus.yml configuration file. This requires a compatible backend endpoint already running and reachable over the network.

# prometheus.yml
remote_write:
  - url: "http://remote-storage-backend:9090/api/v1/write"
    # Optional: queue_config to handle network spikes
    queue_config:
      max_samples_per_send: 1000
      max_shards: 200

Permissions: The Prometheus process must have network egress permissions to the backend port (e.g., 9090). Ensure the backend is configured to accept writes from the Prometheus IP range.

Verifying the Result

To ensure your storage strategy is working, use the following diagnostic checks:

• For Local Storage: Check the --storage.tsdb.retention.time flag in the process list to confirm the active limit.
• For Remote Write: Query the Prometheus internal metrics to ensure samples are actually leaving the server. Run this query in the Prometheus UI:

  prometheus_remote_storage_samples_total

  If the value is increasing, data is being transmitted. If it is stagnant or accompanied by prometheus_remote_storage_samples_failed_total, check your network connectivity and backend logs.

Rollback Procedure

If Remote Write causes excessive CPU load, remove the remote_write block from prometheus.yml and reload the configuration (via SIGHUP or the /-/reload endpoint). This immediately stops the outbound stream without affecting the local TSDB data.