Grafana Provisioning load order constraint for datasources and dashboards
0 reputation · 21 Nov 2024, 13:42 UTC
0 reputation · 21 Nov 2024, 13:42 UTC
Migrating a small application’s monitoring to a config-driven Grafana setup using file provisioning for datasources and dashboards. The goal is a cutover with no visible interruption to viewers and preserved access.
Provisioning of dashboards and datasources is documented for config-driven installs. The operational uncertainty is the ordering and atomicity between the two sets during startup or a config reload. There is no explicit documented constraint on when a dashboard JSON referencing a datasource name is applied relative to the datasource definition, and no documented guarantee that the two provisioning sets are applied atomically.
Does Grafana enforce a deterministic load order between datasources.yaml and dashboards/*.json during a config reload? If a dashboard is processed before its datasource exists, is the dashboard temporarily invalid or rejected? Is there a documented mechanism to make datasource and dashboard provisioning apply as an atomic unit for a zero-downtime migration?
31275 reputation · 21 Nov 2024, 14:29 UTC
Grafana processes datasource provisioning before dashboard provisioning during startup and config reload, at least through v9.4. This order is deterministic in the source code. If a dashboard references a datasource that has not yet been registered, the dashboard loads but displays a "Data source not found" error until the datasource appears, at which point Grafana retries and clears the error automatically. There is no documented mechanism to apply datasource and dashboard changes as a single atomic transaction.
During a reload, viewers may see a brief "Data source not found" banner on affected dashboards. The window is typically seconds, but it is not guaranteed to be invisible. If true zero-downtime is required, you must orchestrate the change outside Grafana (e.g., blue/green instances or a proxy that swaps after both sets are healthy).
logs.level=debug and provisioning.logger=debug in grafana.ini.Loading datasource from … lines appearing before Loading dashboard from ….datasources.yaml. Reload. The dashboard appears with the error banner.The load-order guarantee is confirmed in the Grafana source tree up to v9.4. Behavior in versions <6.0 or in future releases (v10+) is not guaranteed by documentation. If you are running a version outside the 6.0–9.4 range, the order could differ.
One diagnostic detail needed: What Grafana version are you targeting? If it is v10 or later, the recommendation to rely on the built-in order should be re-verified against that release’s changelog or source.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.