Grafana Architecture: Server-Side Proxy and File-Based Provisioning for Trust Boundaries

The Problem: Credentials in the Browser and Drift in Dashboards

When Grafana dashboards query data sources directly from the browser, credentials end up in frontend code or network traces. Teams also struggle to keep dashboards in sync across environments—manual UI edits drift from version control, and secret rotation becomes a game of find-and-replace across multiple repositories. The fix is to move all data source communication behind the Grafana server and treat dashboards as code deployed through file-based provisioning.

Requirements Driving the Design

• No credentials in the browser: Data source passwords, tokens, and TLS certificates must never leave the Grafana host.
• Reproducible dashboard deployment: Dashboards and data sources are defined in YAML/JSON, version-controlled, and applied automatically on startup or reload.
• No direct client-to-backend paths: Firewalls should allow egress only from the Grafana host to approved data sources; browsers talk only to Grafana.

Smallest Suitable Design

A single Grafana instance configured with:

• Server-side data source access (proxy mode): The Grafana server forwards queries to Prometheus, Loki, Tempo, or SQL backends. The browser sees only /api/datasources/proxy/<uid>/... endpoints.
• File-based provisioning: Data sources and dashboards are declared in /etc/grafana/provisioning/datasources/ and /etc/grafana/provisioning/dashboards/. On startup or SIGHUP, Grafana reads these files and creates or updates resources in its internal database.
• Secrets injected at deploy time: Provisioning files reference environment variables (e.g., ${PROMETHEUS_TOKEN}) that a secret manager or CI pipeline populates. Plaintext secrets never touch version control.

Trust and Data Boundaries

Browser → Grafana (Trusted)

The browser authenticates to Grafana (OAuth, LDAP, or Grafana auth). All dashboard queries terminate at Grafana. In browser dev tools, network requests should show the Grafana origin, not the data source host—confirming proxy mode is active.

Grafana → Data Sources (Controlled Egress)

Firewall rules permit outbound connections only from the Grafana host/IP to specific data source endpoints (e.g., Prometheus on 9090, Loki on 3100). No other host in the VPC should reach those ports. This makes Grafana the trust anchor: compromise of a dashboard JSON file does not expose data source credentials.

Credential Storage

Data source secrets live in Grafana's encrypted database (SQLite, Postgres, or MySQL) or an external secret store (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault). Provisioning files contain placeholders:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    uid: prom-prod
    access: proxy
    url: https://prometheus.internal:9090
    jsonData:
      tlsSkipVerify: false
    secureJsonData:
      httpHeaderValue1: ${PROMETHEUS_BEARER_TOKEN}

At deploy time, your pipeline or init container replaces ${PROMETHEUS_BEARER_TOKEN} with the actual value from the secret store. The rendered file is written to the provisioning directory; Grafana reads it and stores the secret in secureJsonData (encrypted at rest).

Operational Checks

1. Provisioning Health at Startup

After Grafana starts, verify in the UI:

• Configuration → Data sources: each provisioned source shows a green “Test successful” badge.
• Dashboards → Manage: provisioned dashboards appear under their folder with a lock icon (read-only).

Tail logs for parse errors:

# On the Grafana host (requires access to container or systemd journal)
journalctl -u grafana-server -f | grep -i provisioning
# or in Kubernetes
kubectl logs -n monitoring deploy/grafana -c grafana | grep -i provisioning

Expected: lines like msg="Provisioning data sources" and msg="Dashboard provisioning completed". Errors surface as level=error msg="Failed to provision" with YAML line numbers.

2. Data Source Connectivity Test

Use the built-in health endpoint programmatically:

curl -s -H "Authorization: Bearer $GRAFANA_API_TOKEN" "$GRAFANA_URL/api/datasources/uid/prom-prod/health"
# Expected: {"status":"ok","message":"Data source is working"}

Run this in a cron job or Prometheus blackbox exporter to alert on backend loss.

Failure Modes and Mitigations

Failure Mode	Symptom	Mitigation
Malformed provisioning YAML	Grafana fails to start or skips the bad file; dashboards/data sources missing	Validate YAML in CI (yamllint, grafana-toolkit); test render in staging before prod deploy
Data source outage	Panels show “Error querying data source”; health endpoint returns non-ok	Alert on health endpoint; configure datasource timeout and maxConcurrentRequests in provisioning
Excessive data source permissions	Team A sees Team B's metrics because shared Prometheus token has global scope	Use separate data source UIDs per team with scoped tokens; enforce via Grafana folder permissions
Environment drift	Provisioning works in dev but fails in prod due to missing env vars or API version mismatch	Pin Grafana version in Dockerfile/Helm chart; use same provisioning schema version across envs

Critical Cautions

• Provisioned resources are overwritten on reload. Manual UI edits to a provisioned dashboard or data source are lost when Grafana re-reads the files. Treat the UI as read-only for those objects; make changes in the provisioning repo and redeploy.
• Proxy mode does not enforce row-level security. Grafana forwards the query as-is. If Prometheus has no tenant isolation, a dashboard variable can leak another team's labels. Row-level security must be implemented in the data source (e.g., Prometheus remote-write with tenant headers) or via Grafana Enterprise's data source permissions.
• Anonymous access bypasses trust boundaries. If auth.anonymous.enabled = true and a dashboard is public, anyone with the URL can query proxied data sources. Disable anonymous access or restrict public dashboards to read-only, non-sensitive data sources.

Conditions That Change the Design

• High availability: Multiple Grafana replicas require a shared database (Postgres/MySQL) and consistent provisioning files across nodes. Use a configmap or shared volume with a reload sidecar (e.g., kiwigrid/k8s-sidecar) to propagate file changes.
• Strict multi-team isolation: Grafana Enterprise's data source permissions and teams/folders provide RBAC. Without Enterprise, run separate Grafana instances per team or use a reverse proxy with authz checks.
• Heavy query volume: Add query caching (Grafana's built-in cache or Redis backend) and consider read replicas for SQL data sources. For Prometheus, use recording rules and subquery optimization rather than scaling Grafana.
• Regulated secret rotation: If secrets must rotate without Grafana restart, integrate an external secrets operator (e.g., external-secrets-operator) that updates the provisioning file and sends SIGHUP, or use Grafana's HTTP API to update secureJsonData directly.

Verification Checklist Before Exposing Grafana

1. Proxy confirmation: Open a dashboard in the browser, open DevTools Network tab, run a query. Request URL must be https://grafana.yourdomain.com/api/datasources/proxy/uid/..., not the data source host.
2. Provisioning error visibility: Temporarily corrupt a provisioning file (e.g., remove a required field), restart Grafana, and verify logs show a clear parse error and the instance does not silently serve stale dashboards.
3. Credential revocation test: Use the data source test action in the UI (green badge). Revoke the underlying credential (rotate token in Vault, disable DB user). Repeat the test; confirm failure is logged and surfaced in the UI rather than returning empty panels.
4. Firewall validation: From a host other than Grafana, attempt curl prometheus.internal:9090. Connection must be refused or timeout.

Limitations

• This design assumes a single Grafana instance or HA pair with shared storage. It does not cover multi-cluster federation or Grafana Cloud managed service specifics.
• File-based provisioning does not support dynamic dashboard templating at runtime (e.g., per-user variables stored in the dashboard JSON). For that, use the HTTP API or Grafonnet/Jsonnet to generate dashboards before provisioning.
• Secret injection via environment variables works for static secrets. For short-lived tokens (e.g., cloud IAM tokens with 1-hour TTL), you need a sidecar that rewrites the provisioning file and signals reload, or use the Grafana API directly.

Practical Result Check

After deploying, run this one-liner from your workstation to confirm the full chain:

curl -s -H "Authorization: Bearer $GRAFANA_API_TOKEN" "$GRAFANA_URL/api/datasources/uid/prom-prod/health" | jq -r .status
# Should print: ok

If it returns ok, your provisioning rendered correctly, the secret was injected, the proxy path is open, and the data source is reachable. That's the signal that the trust boundary holds.