Configure Prometheus Remote Write with TLS and Basic Authentication

Problem and takeaway

Prometheus scrapes metrics locally by default. If you need to store those metrics in a remote system—such as a managed time-series database or a central aggregator—you must configure remote write. Sending that traffic without TLS exposes both the metrics and any basic-auth credentials in plaintext on the network.

The useful takeaway: add a remote_write block to prometheus.yml, enable TLS with tls_config, supply basic auth credentials, then validate with promtool and remote-storage metrics before relying on the path in production.

Desired outcome

Prometheus forwards scraped samples to a remote HTTPS endpoint over a TLS-encrypted connection. The remote server authenticates Prometheus with basic auth, and Prometheus verifies the server’s certificate against a trusted CA. If client certificates are provided, both sides authenticate each other (mutual TLS).

Prerequisites

• Prometheus 2.30 or later. Earlier releases do not support min_version and cipher_suites inside tls_config, and using those fields causes startup errors.
• Access to the Prometheus configuration file, typically /etc/prometheus/prometheus.yml.
• A CA certificate file (ca.crt) for the remote endpoint. Optional client certificate (client.crt) and key (client.key) if the endpoint requires mutual TLS.
• A username and password for the remote write endpoint.
• Network reachability from the Prometheus host to the remote endpoint on port 443 or the configured HTTPS port.

Procedure

1. Place certificate files on the Prometheus host

Copy the CA certificate and, if required, the client certificate and key to a directory readable by the Prometheus process, for example /etc/prometheus/certs/. Restrict permissions so only the Prometheus user can read the private key:

sudo cp ca.crt client.crt client.key /etc/prometheus/certs/
sudo chown -R prometheus:prometheus /etc/prometheus/certs/
sudo chmod 600 /etc/prometheus/certs/client.key

Run these commands on the Prometheus server with root or sudo privileges. Incorrect file ownership causes Prometheus to fail reading the certificate at startup.

2. Add the remote_write block

Open prometheus.yml and add a remote_write block under the top-level remote_write: key. Replace placeholders with your actual endpoint and credentials:

remote_write:
  - url: https://remote.example.com/api/v1/write
    basic_auth:
      username: remote_user
      password: remote_password
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      cert_file: /etc/prometheus/certs/client.crt
      key_file: /etc/prometheus/certs/client.key
      min_version: TLS1.2
    queue_config:
      capacity: 10000
      max_samples_per_send: 500
      max_shards: 20

URL-encode special characters in the username or password. Unencoded characters can produce 401 errors and fill the remote write queue.

3. Validate the configuration

Run Prometheus’s built-in config checker before reloading:

promtool check config /etc/prometheus/prometheus.yml

Run this on the Prometheus host as the prometheus user or a user with read access to the config. A successful check prints no errors and exits with code 0.

4. Reload Prometheus

If validation passes, reload Prometheus so it picks up the new remote write target:

sudo systemctl reload prometheus

If systemd is not used, send SIGHUP to the Prometheus process:

sudo kill -HUP $(pgrep prometheus)

Expected checks

Check Prometheus logs

After reload, inspect the Prometheus log for confirmation that remote storage initialized and that no TLS handshake errors appear:

journalctl -u prometheus -n 50 --no-pager

Look for a message indicating remote storage is enabled. Absence of tls: handshake failure or certificate errors means the TLS configuration is likely valid.

Test the endpoint manually

From a host that can reach the remote endpoint, run a manual TLS request with the same credentials and certificates:

curl -v -u remote_user:remote_password \
  --cert /etc/prometheus/certs/client.crt \
  --key /etc/prometheus/certs/client.key \
  --cacert /etc/prometheus/certs/ca.crt \
  https://remote.example.com/api/v1/write

Expect a 204 No Content or similar success response from a compatible remote write API. A 401 means the credentials are wrong or not URL-encoded. A TLS alert means the CA or certificate chain is mismatched.

Confirm sample delivery

Query Prometheus itself for remote write metrics. While scraping targets, the counter prometheus_remote_storage_sent_samples_total should increase. The gauge prometheus_remote_storage_queue_length should remain low or return to zero after bursts. Sustained growth means the remote endpoint cannot keep up and Prometheus is buffering samples in memory.

prometheus_remote_storage_sent_samples_total
prometheus_remote_storage_queue_length
prometheus_remote_storage_failed_samples_total

Limitations

• Basic auth credentials are stored in plaintext in prometheus.yml. Protect the file with restrictive permissions (for example, chmod 600 owned by prometheus).
• Basic auth without TLS sends credentials in cleartext. Never use basic_auth with an http:// URL.
• The remote write queue consumes memory. If the remote endpoint is slow or down, Prometheus buffers samples according to queue_config. Tune capacity, max_samples_per_send, and max_shards based on available memory and scrape volume.
• Prometheus versions before 2.30 reject min_version and cipher_suites in tls_config. Omit those fields on older releases.

Recovery options

This procedure changes Prometheus state by adding a remote write target. To revert, remove the remote_write block from prometheus.yml, run promtool check config, and reload Prometheus. Pending samples in the queue are discarded on reload if the remote target is removed.