Provision Grafana Dashboards from Kubernetes ConfigMaps Using Declarative JSON

To keep Grafana dashboards under version control and have them appear automatically in a Kubernetes deployment, mount a ConfigMap that contains dashboard JSON files to Grafana’s provisioning directory and configure a file‑based provider with a short updateIntervalSeconds. Grafana will then load, update, and keep the dashboards in sync without requiring a pod restart.

Worked Example: grafana.ini, ConfigMap, and Dashboard JSON

1. Configure the file provider in grafana.ini

Add a [dashboards] section that points to the directory where the ConfigMap will be mounted. The provider type is file; set options.folder to the desired Grafana folder (or leave empty for "General") and enable automatic reloads.

[dashboards.provisioning]
enabled = true
path = /var/lib/grafana/dashboards

[dashboards]
# This block is read from files under the path above; we define one provider via a file.
# Example file: /var/lib/grafana/dashboards/provisioning/dashboards.yaml

Create the provider file dashboards.yaml inside the mounted directory:

apiVersion: 1
providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10   # reload every 10 seconds
    options:
      path: /var/lib/grafana/dashboards/default
      allowUiUpdates: false

2. Define a minimal dashboard JSON

Each dashboard must contain a stable uid, a title, a version (used for update ordering), schemaVersion, and a panels array. Remove any id field exported from the UI.

{
  "uid": "my-service-dashboard",
  "title": "Service Metrics",
  "version": 1,
  "schemaVersion": 38,
  "panels": [
    {
      "type": "graph",
      "title": "Request Rate",
      "datasource": { "type": "prometheus", "uid": "PBFA97CFB590B2093" },
      "targets": [ { "expr": "rate(http_requests_total[5m])", "legendFormat": "{{instance}}" } ]
    }
  ]
}

3. Package dashboards in a ConfigMap and mount them

Place the JSON file (e.g., my-service-dashboard.json) in a directory that matches the provider’s options.path. Then create a ConfigMap that contains the file(s).

# Directory layout before creating the ConfigMap
# dashboards/
# └── default/
#     └── my-service-dashboard.json

Create the ConfigMap (run with sufficient RBAC to edit ConfigMaps in the target namespace):

kubectl create configmap grafana-dashboards \
  --from-file=dashboards/default/my-service-dashboard.json \
  -n monitoring

Mount the ConfigMap into the Grafana pod at the provider’s path:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  template:
    spec:
      containers:
        - name: grafana
          volumeMounts:
            - name: dashboards
              mountPath: /var/lib/grafana/dashboards/default
              readOnly: true
      volumes:
        - name: dashboards
          configMap:
            name: grafana-dashboards
            items:
              - key: my-service-dashboard.json
                path: my-service-dashboard.json

How Provisioning Works

Grafana reads all [dashboards] provider files from the directory defined in grafana.ini. For each provider it scans the configured options.path every updateIntervalSeconds seconds. When a file is found:

• Grafana extracts the uid. If a dashboard with that uid already exists, it compares the incoming version with the stored version; the dashboard is replaced only if the incoming version is higher.
• If no matching uid exists, a new dashboard is created.
• The dashboard is placed in the folder specified by options.folder (or the folder looked up by options.folderUid). If omitted, it lands in "General".
• With allowUiUpdates: false (default) the UI edit button is disabled, ensuring the source‑controlled JSON remains the single source of truth.

Limits and Common Mistakes

• Version handling: Grafana only overwrites when the incoming version is numerically higher. Forgetting to increment version in CI results in no UI changes despite file edits. Fix: automate version bumps or rely solely on file mtime checks by setting a low updateIntervalSeconds and ensuring the file’s modification time changes on each commit.
• Missing or wrong uid: Committing a dashboard exported from the UI often leaves the numeric id field and omits uid. Grafana ignores id on import, causing a new dashboard to be created on every provisioning pass (duplicates). Fix: strip id and generate a stable uid (e.g., using grafana-cli admin update-dashboard-uid on an existing dashboard).
• Folder misconfiguration: If options.folder refers to a non‑existent folder, dashboards fall back to "General". Use options.folderUid with the UID of a pre‑created folder, or create the folder via the API before provisioning.
• Allowing UI updates: Setting allowUiUpdates: true creates a fork that diverges from the source‑controlled JSON; subsequent provisioning will not overwrite UI‑made changes, leading to drift. Keep this flag false unless you intentionally want editable copies.
• Datasource references: Hard‑coding datasource names (e.g., "datasource": "Prometheus") breaks when the same dashboard is applied to a cluster with a different datasource name. Use the uid form or template variables like ${DS_PROMETHEUS}.
• Plugin version fields: Including pluginVersion in panel JSON can cause rendering failures if the plugin version differs between environments. Omit the field or ensure plugin parity across clusters.
• Large dashboard sets: Hundreds of JSON files increase startup time and memory consumption. Split dashboards across multiple providers with distinct folders or use the HTTP API/Terraform provider for very large inventories.
• Managed services: Grafana Cloud and some managed offerings disable filesystem provisioning. In those cases, use the Grafana HTTP API or the official Terraform provider instead of ConfigMaps.

Verification Steps

1. Check provider configuration: Exec into the Grafana pod and view the effective ini. kubectl exec -n monitoring deploy/grafana -- cat /etc/grafana/grafana.ini | grep -A2 '\[dashboards\]' Look for the path pointing to /var/lib/grafana/dashboards and confirm a provider file exists under that directory.
2. Inspect the ConfigMap: kubectl get configmap grafana-dashboards -n monitoring -o yaml Verify that each key is a valid JSON file containing a uid and no id field.
3. View loaded providers in the UI: Navigate to Settings → Provisioning → Dashboards. The provider should list default with a Last refreshed timestamp that updates roughly every updateIntervalSeconds seconds.
4. Test an update: Modify the dashboard JSON (e.g., change a panel title), increment the version field, and apply the updated ConfigMap (kubectl apply -f configmap.yaml). Wait at least updateIntervalSeconds seconds, then:
   • Open the dashboard in Grafana UI and confirm the change appears.
   • Or query the HTTP API: curl -s -H "Authorization: Bearer $TOKEN" http://grafana/api/dashboards/uid/my-service-dashboard | jq .dashboard.title (replace $TOKEN with a valid API key). The returned title should match your edit.
   If the edit button remains disabled, allowUiUpdates: false is working as intended.
5. Validate JSON schema (optional): If the grafana-cli binary is available in the pod, run: grafana-cli admin verify-dashboard /var/lib/grafana/dashboards/default/my-service-dashboard.json A successful validation returns no error; malformed JSON will produce a clear message.
6. Check for duplicates: Use the API to list all dashboards with the same uid: curl -s -H "Authorization: Bearer $TOKEN" http://grafana/api/search?uid=my-service-dashboard | jq . Expect exactly one entry.