Kibana and Elasticsearch upgrade order: which component to upgrade first on a production cluster
0 reputation · 03 Jul 2024, 10:42 UTC
0 reputation · 03 Jul 2024, 10:42 UTC
We run a Kibana instance connected to a self-managed Elasticsearch cluster, both currently on the same minor version of a recent 8.x release. Elastic's documentation states that Kibana and Elasticsearch are expected to run matching versions, with only narrow exceptions during rolling upgrades, and that Kibana newer than Elasticsearch is unsupported because of versioned saved-object metadata in the system indices.
That leaves an unresolved sequencing decision for our next minor upgrade. Upgrading Elasticsearch first means Kibana runs temporarily older than the cluster, which is described as transitional rather than a supported steady state. Upgrading Kibana first appears to risk startup compatibility-check failures or saved-object migration problems against the older cluster. Coordinating both at once implies a short full outage for dashboards.
Upgrade Elasticsearch first, then Kibana, to the same minor version. This is the documented and safer order for a minor upgrade within the same major release. Kibana is designed to work with a range of Elasticsearch versions, but Elasticsearch does not guarantee backward compatibility with an older Kibana. Running a newer Kibana against an older Elasticsearch is unsupported and will cause startup failures or saved-object migration errors.
During a rolling Elasticsearch upgrade, your cluster temporarily runs a mix of old and new nodes. Kibana, still on the old version, can continue to serve dashboards because it tolerates a newer Elasticsearch within the supported version range. Once all Elasticsearch nodes are upgraded, you upgrade Kibana. The reverse order—Kibana first—risks Kibana refusing to connect to the older cluster or attempting saved-object migrations against incompatible system indices.
curl -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster.routing.allocation.enable":"none"}}'curl -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster.routing.allocation.enable":null}}'
curl 'localhost:9200/_cluster/health?wait_for_status=green'curl 'localhost:9200'
curl 'localhost:5601/api/status'Hard stops—treat these as blocking errors, not warnings:
This version of Kibana is incompatible with Elasticsearch.Unable to complete saved object migrations.incompatible version errors.Tolerable transitional warnings during the Elasticsearch-first window:
If Kibana migrates its saved objects during startup, there is no supported path back to the older Kibana version. Snapshot and restore of the system indices (e.g., .kibana*) across versions is not reliable because the saved-object metadata format changes. Plan your upgrade window accordingly: once Kibana starts and completes migration, rolling back requires restoring the entire cluster from a pre-upgrade snapshot taken before any component was upgraded, and even then, only if you also downgrade Elasticsearch. In practice, treat the upgrade as one-way after Kibana begins its migration.
For exact version compatibility, consult the official Elastic Stack compatibility matrix for your source and target versions, and read the upgrade guide for your specific version jump—especially if you are crossing a major version, which may require additional steps like reindexing.
Use comments to ask for clarification. Post a solution as an answer.
2,320 reputation · 03 Jul 2024, 21:06 UTC
When performing a minor version bump within the same 8.x major line, upgrade the Elasticsearch cluster first using a rolling restart. During the roll‑out Kibana (still on the old version) can continue to query the mixed‑version cluster because it remains backward‑compatible with newer Elasticsearch nodes. Only after all Elasticsearch nodes report the new version and cluster health is green should Kibana be stopped, upgraded, and started. This avoids the startup compatibility check that would reject a newer Kibana talking to an older Elasticsearch and prevents Kibana from attempting saved‑object migrations against incompatible system indices.
Use the Kibana Upgrade Assistant (Management → Stack Management → Upgrade Assistant) to confirm that the Elasticsearch version meets the minimum required for the target Kibana version before proceeding.