Indexer Cluster Rolling Upgrade and Long-Running Search Continuity

Answer

Splunk Enterprise 9.x does **not** automatically redirect an active search job to another indexer when the node hosting that job is taken offline during a rolling upgrade. The running job will terminate unless the search head re‑submits it to a peer.

Confirmed facts

• Search jobs executing on a stopped indexer are not migrated; they end with a "search job terminated" message.
• Continuity depends on the cluster’s replication factor (RF) and search factor (SF).
• If RF ≥ 2 and SF ≥ 1, a search head can route a new (or re‑submitted) job to a replica on another active indexer, provided the job has not exceeded its maximum execution time.
• Thus, to avoid a timeout caused by a single node going offline, you need at least one additional active indexer that holds a copy of the data – i.e., RF > 1.

Steps for this case

1. Check the current replication factor: splunk show cluster-status (look for "replication_factor").
2. Verify the search factor (usually defined in the indexer cluster master server.conf under [clustering] or via the UI).
3. If RF = 1, increase it to at least 2 before the upgrade (adds a copy of each bucket).
4. Ensure SF is at least 1 (default is 1); a higher SF (e.g., 2) provides more search‑head redundancy.
5. Schedule the rolling upgrade during low‑search‑activity windows and monitor job logs for termination messages.
6. If a long‑running job is critical, consider breaking it into smaller chunks or using saved searches with checkpoints that can be resumed.

Missing diagnostic detail

To give a precise threshold for your environment, I need to know the current replication factor and search factor of your indexer cluster. Please provide those values so I can confirm whether the existing setup meets the RF > 1 requirement.