Question
Snapshot restore behavior change in Elasticsearch 7.0+ affecting post‑restore cluster health
Lax Nova
0 reputation · 07 Mar 2023, 01:02 UTC
46.4K views
Loading…
0 reputation · 07 Mar 2023, 01:02 UTC
Goal: confirm that a restored index contains all expected documents without waiting for the restore operation to finish, while accepting that replica shards may remain unallocated and cluster health may show yellow.
Constraints: the restore API returns immediately by default in Elasticsearch 7.0+, and verification queries must not interfere with ongoing shard allocation; it is unclear whether a match‑all request issued right after the API response reliably reflects the restored data or whether it should be delayed until replica allocation completes.
Questions:
• Is it safe to execute a match‑all query immediately after the restore call returns, or should verification wait for a specific cluster state?
• Can yellow health caused by unallocated replicas be tolerated for verification purposes, or must the cluster be green before checking document counts?
• Does explicitly setting wait_for_completion=true change the verification timing or guarantee that the index is fully queryable right after the restore finishes?
35775 reputation · 07 Mar 2023, 08:24 UTC
Yes – you can run a match_all (or _count) request as soon as the restore API returns, provided the primary shards of the index have finished recovery. The cluster may stay yellow because replica shards are still allocating; this does not affect read correctness.
elasticsearchwait_for_completion=false and returns after initiating the restore.stage: DONE in the index recovery API.wait_for_completion=true blocks the call until both primaries and replicas are done, guaranteeing a green health on return, but it does not make data visible earlier than the asynchronous path.wait_for_completion=false).GET /_snapshot///_status if you need to know when it finishes.GET //_recovery?active_only=true until every primary shard shows "stage":"DONE" and "primary":true.GET //_count or a match_all with size=0.GET /_cluster/health) as normal; you do not need to wait for green.If the restored index has been snapshotted with blocking settings (for example index.blocks.read_only or other index.blocks flags), those settings are restored and will reject queries even though primaries are recovered. In that case you must clear the blocks before verification. Please confirm whether the restored index has any index‑level blocks set.
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 07 Mar 2023, 06:02 UTC
In 7.x the restore API returns as soon as the *primary* shards start recovery. The index becomes queryable once those primaries report stage=DONE, even if replicas are still unassigned. A match_all query issued immediately after the API call will see the full document set as long as the primaries are finished.
To avoid accidental partial counts, poll the cluster health or recovery status instead of guessing:
POST /_snapshot/repo/snap/_restore
{
"indices": "my_index",
"wait_for_completion": false,
"wait_for_active_shards": "all"
}
# Wait until primaries are ready
GET /my_index/_health?wait_for_status=yellow
# Or check recovery stage
GET /_cat/recovery?v&active_only=true
Cluster health can stay yellow during this phase; yellow indicates replicas are unassigned but the primary data is available. Setting wait_for_completion=true blocks until replica allocation completes, which guarantees a green cluster but adds unnecessary latency for simple data verification.