Using Grafana Provisioning to Keep Dashboards and Data Sources in Sync Across Environments

The problem: drifting Grafana configurations

When you manage multiple Grafana instances (dev, staging, production) it’s easy for dashboards and data sources to diverge. A tweak made in the UI on one server can be forgotten when you spin up a new environment, leading to inconsistent monitoring and wasted debugging time.

Thesis: provisioning as code eliminates drift

Grafana’s provisioning feature lets you define dashboards, data sources, and notification channels in plain files that Grafana reads on startup. By treating those files as version‑controlled artifacts, you guarantee that every instance starts with the exact same set of resources.

How provisioning works

You tell Grafana where to look for provisioning files via grafana.ini (or environment variables). The key sections are:

[paths]
provisioning = /etc/grafana/provisioning

Under that directory Grafana expects subfolders matching the resource type:

• dashboards/ – JSON dashboard definitions
• datasources/ – YAML or JSON data‑source definitions
• notification/ – notification channel definitions (optional)

Each file must follow the Grafana API schema for the resource it defines. When Grafana starts (or when the provisioning system is reloaded via grafana-cli admin provisioning reload), it loads the files, creates or updates matching resources, and logs messages such as “Loading dashboards from …”.

Worked example: provisioning a Prometheus data source and a dashboard

Assume you want every Grafana instance to have a Prometheus data source called prometheus-prod and a dashboard named Node Exporter Full stored in a folder called Infrastructure.

1. Data‑source file

Create /etc/grafana/provisioning/datasources/prometheus.yaml:

apiVersion: 1

datasources:
  - name: prometheus-prod
    type: prometheus
    access: proxy
    url: http://prometheus.example.com:9090
    isDefault: true
    editable: false

The editable: false flag prevents users from changing the data source via the UI; any manual edit will be overwritten on the next provisioning reload.

2. Dashboard file

First, place the dashboard JSON (you can export it from an existing Grafana instance) under /etc/grafana/provisioning/dashboards/infrastructure/. For example, node_exporter.json:

{
  "dashboard": {
    "id": null,
    "uid": "node-exporter-full",
    "title": "Node Exporter Full",
    "tags": ["infrastructure", "node-exporter"],
    "timezone": "browser",
    "schemaVersion": 38,
    "version": 1,
    "refresh": "10s",
    "panels": []
  },
  "folderId": 0,
  "overwrite": true
}

Note the overwrite: true flag; it tells Grafana to replace any existing dashboard with the same UID.

3. Folder provisioning (optional)

If you want the dashboard to appear inside a specific folder, add a folder provisioning file /etc/grafana/provisioning/dashboards/folder.yaml:

apiVersion: 1

providers:
  - name: 'infrastructure'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/infrastructure

This makes Grafana treat the infrastructure directory as a source for dashboards and creates (or updates) a folder named “Infrastructure” automatically.

Trade‑off: provisioned resources can be overwritten

The biggest operational consideration is that any change made through the UI that conflicts with the provisioned definition will be lost on the next Grafana restart or provisioning reload. Teams often adopt one of two strategies:

• Treat provisioned files as the source of truth and disable UI edits for those resources (editable: false for data sources, allowUiUpdates: false for dashboards).
• Allow UI tweaks for exploratory work, then periodically export the updated dashboard or data source back into the provisioning repository.

Another practical limitation is startup time: loading hundreds of large dashboard JSON files can add seconds to Grafana’s boot. If you manage a very large catalog, consider:

• Splitting dashboards into multiple provisioning folders and using updateIntervalSeconds to reload only changed folders.
• Using external plugins or side‑car services that fetch dashboards on demand rather than provisioning them all at startup.

Actionable steps to verify your setup

1. Place the provisioning files as shown above.
2. Start Grafana (or run grafana-cli admin provisioning reload if it’s already running).
3. Check the logs for lines like:

logger=provisioning.dashboard level=info msg="Loading dashboards from /etc/grafana/provisioning/dashboards"

logger=provisioning.datasource level=info msg="Loading data sources from /etc/grafana/provisioning/datasources"

4. Open the Grafana UI:
• Navigate to Configuration → Data Sources and confirm prometheus-prod appears and is set as default.
• Go to Dashboards → Manage and verify the Infrastructure folder contains the Node Exporter Full dashboard.
5. To test overwrite behavior, edit the dashboard title in the UI, save, then restart Grafana. After restart the title should revert to the value in node_exporter.json.

By treating dashboards and data sources as code, you gain reproducible environments, easier rollback via Git, and a clear audit trail of what each Grafana instance should contain.