Grafana zero-downtime data source migration: file-based provisioning reload or HTTP API partial updates?

Answer

For a single‑node Grafana 10.x instance the cleanest, safest cutover is to use the /api/datasources/:id endpoint with a single PUT that replaces the entire datasource definition. The API performs the update in one transaction, so the URL and credentials change atomically and dashboards can keep querying the same UID without a visible gap.

Why the API is safer than file‑based provisioning

• Atomicity: The API applies the whole JSON payload at once. If the request fails, the datasource stays unchanged.
• No reload pause: Provisioning reloads all datasources in the file, briefly disabling queries until the reload finishes.
• Visibility: You can see the exact state in the response and in the Grafana UI immediately after the call.

How provisioning and the API interact

• Datasource objects exist in two places: the database (created/updated via the API) and optional provisioning files. When a provisioning file is reloaded, Grafana overwrites the database entries that match the file’s UID.
• Therefore, if a datasource is present in both a provisioning file and the database, any subsequent file reload will undo API changes.
• To keep the API state intact, exclude the datasource from the provisioning file (comment it out or delete it) before performing the migration.

Recommended cutover steps

1. Back up the current datasource JSON:

   curl -s -H "Authorization: Bearer $TOKEN" "http://grafana/api/datasources/:id" > ds.json

2. Validate the payload by editing ds.json with the new URL and credentials.
3. Test the update with a dummy URL to confirm that a failing request leaves the datasource unchanged.
4. Apply the real migration:

   curl -s -X PUT \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d @ds.json \
     http://grafana/api/datasources/:id

5. Immediately query a panel that uses the datasource to verify success.
6. Optional: if you prefer to keep the old UID, create a new datasource via the API, switch dashboards to the new UID in a rolling fashion, then delete the old one.
7. If you were using provisioning, remove the datasource from the YAML file before the migration and re‑enable it after the API change is verified.

Atomicity caveat

Partial field updates (e.g., only the URL) via the API are not atomic; they can leave the datasource in a half‑updated state if the request is interrupted. Always send the full JSON payload.

What we still need to know

Do any dashboards reference the datasource via a UID that is shared with other instances (e.g., a shared datasource across multiple Grafana servers)? Knowing this will confirm whether a single‑UID update is sufficient or if a new UID path is required.