Silent document drift during Elasticsearch Reindex API migration

Answer

To guarantee that no writes are lost during a reindex, route every write to both the source and the target indices (dual‑write) and perform a quick final sync after the reindex completes. This is the most reliable strategy for most applications. If dual‑write is not feasible, use an external change‑feed (Kafka, Logstash, etc.) to replay changes after the initial reindex.

Likely Explanation

The Reindex API captures a snapshot of the source index at the moment the request starts. Any subsequent creates, updates, or deletes are invisible to that snapshot. Without an explicit mechanism to propagate those changes, the target index will drift silently.

Confirmed Steps for This Case

1. Create a PIT on the source index

   PUT /source_index/_pit?keep_alive=1m
   { "keep_alive": "1m" }
   

2. Start the reindex operation

   POST /_reindex
   {
     "source": {
       "index": "source_index",
       "pit": { "id": "" }
     },
     "dest": { "index": "target_index" }
   }
   

3. Route all writes to both indices
   • Application‑level proxy or interceptor that performs the same index, update, or delete on source_index and target_index.
   • Ensure idempotency: use op_type: "index" with the same document id, or op_type: "create" plus a retry loop.
   • Log failures per index and surface them to the operator for manual replay if needed.
4. When reindex finishes, perform a consistency check
   • Compare document counts:

     GET /source_index/_count
     GET /target_index/_count
     

   • Random sample test: retrieve 100 random ids from source_index and query target_index for each.

     GET /source_index/_search
     {
       "size": 100,
       "query": { "function_score": { "random_score": {} } }
     }
     

   • Use the _reindex/_stats endpoint to confirm total equals rejected + updated + created and that failures are zero.
5. Perform a short final sync
   • Pause the dual‑write proxy for 1–2 seconds to allow any in‑flight writes to finish.
   • Optionally run a small “catch‑up” bulk reindex of the last few seconds’ worth of data using a time‑based query.
6. Atomically switch the alias

   POST /_aliases
   {
     "actions": [
       { "remove": { "index": "source_index", "alias": "app_data" } },
       { "add":    { "index": "target_index", "alias": "app_data" } }
     ]
   }
   

7. Stop routing writes to the old index
   • Remove the dual‑write proxy or adjust the application to write only to target_index.
   • Verify that new writes are reflected immediately in search results.

Verification Checklist

• _count on source and target match.
• No missing IDs in the random sample test.
• Zero failures reported by _reindex/_stats.
• Search queries against the alias return identical results before and after the switch.

Missing Diagnostic Detail Needed to Refine the Recommendation

What is the average write throughput (documents per second) during the migration window? Knowing this will help decide if dual‑write is viable or if an external change‑feed is required.