Grafana data source TLS: limits of CA certificate reloads and DNS caching without restart

Question

GoalI am operating Grafana (assume a current LTS release) with several provisioned data sources (Prometheus and Loki) that use TLS with a custom CA supplied via tls_ca. I want to understand the operational limits of certificate validation and DNS resolution for these connections, specifically around rotation and failover scenarios.Constraints and uncertaintyMy understanding is that Grafana validates data source certificates against the system trust store unless tls_ca is set, and that hostname resolution depends on the host OS resolver plus Go HTTP client connection reuse. Two behaviors are unclear to me: whether updating the CA file referenced by tls_ca takes effect without restarting Grafana or re-saving the data source, and how long stale DNS answers can persist for a data source hostname after the record changes, given TCP connection reuse.QuestionsIs there any supported mechanism (API call, provisioning reload, or signal) that makes Grafana pick up a changed tls_ca file for existing data sources without a full restart?Does Grafana's HTTP client honor DNS TTLs for data source connections, or can pooled connections keep using an old IP indefinitely until restart?Is certificate revocation checking (CRL/OCSP) performed at all for data source TLS connections, or is it entirely out of scope?

Tasadduq Burney · Answer

Grafana data source TLS: CA reload and DNS caching limits

Below are the confirmed behaviors for a current Grafana LTS release, based on the Go runtime it uses.

1. Custom CA certificate reload

Grafana does not monitor the tls_ca file for changes. The CA is loaded when the data source is first provisioned or when it is saved via the UI or API. Updating the underlying file without re-saving the data source or restarting Grafana has no effect on active connections.

2. DNS resolution and connection pooling

Grafana's HTTP client relies on the OS resolver and Go's net/http package, which does not honor DNS TTLs for pooled connections. A connection that resolves a hostname once may reuse the resulting IP address for the duration of the idle-connection timeout (default 90s), even if the DNS record changes. The timeout is configurable via connection_idle_timeout in grafana.ini.

3. Certificate revocation checking

Grafana does not perform CRL or OCSP revocation checking for data source TLS handshakes. It only verifies the certificate chain against the supplied CA and checks the validity period. If revocation status is critical, it must be handled externally (e.g., via a proxy or CA configuration that enforces online checks).

Verification scoped checks

Edit the data source's tls_ca field via the UI or API, then restart Grafana or re-save the data source. Confirm the new certificate chain is accepted by checking Grafana logs for TLS handshake entries.
Point a data source hostname to one IP, then update the DNS record to a different IP. Issue several requests and observe the IP address used; connections will persist on the resolved IP until the idle-connection timeout elapses.
Check Grafana logs for any CRL/OCSP-related messages during a TLS handshake; their absence confirms that revocation checking is not performed.

(Note: if you have set a custom connection_idle_timeout in grafana.ini, its value will affect how long stale DNS answers can persist.)

Grafana data source TLS: limits of CA certificate reloads and DNS caching without restart

Goal

Constraints and uncertainty

Questions

1 answer

Grafana data source TLS: CA reload and DNS caching limits

1. Custom CA certificate reload

2. DNS resolution and connection pooling

3. Certificate revocation checking

Verification scoped checks

Your perspective belongs here.

0 question comments