Couchbase Upgrade: Indexes Stuck in 'Indexing' State During Upgrade

Question

During a Couchbase Server upgrade, N1QL indexes sometimes remain in the indexing state, causing query timeouts. This symptom suggests that the upgrade process may not complete the necessary rebuilds automatically.
Key uncertainties: the upgrade logs frequently omit explicit completion timestamps for index rebuilds; it is not clear if all index types are rebuilt automatically or if some require manual reindexing; cluster health metrics may still report healthy while indexes are not ready.
The goal is to determine the exact behavior of the upgrade process regarding index rebuilds and to identify reliable indicators that an index is fully rebuilt.
What specific log entries or API responses confirm that an index rebuild has finished during or after a Couchbase Server upgrade? Does the upgrade automatically trigger a full rebuild for all N1QL index types, or are manual steps required for certain types? How can we programmatically verify that all indexes report ready before resuming normal traffic?

Tasadduq Burney · Answer

Direct answer

A Couchbase Server upgrade does not automatically rebuild every N1QL Global Secondary Index. Index definitions and their data normally persist across an upgrade, so an index already in indexing stays there until the indexer finishes the build or you manually rebuild it. Cluster health reporting healthy is not proof that indexes are ready.

Confirmed indicators versus uncertain signals

Confirmed: system:indexes state changing from indexing to ready for the affected index.
Confirmed: indexer stats showing zero active builds after the build completes.
Not reliable alone: cluster or service health, absence of upgrade log messages, or a single system:indexes check.

Steps for this case

Run SELECT name, keyspace_id, state FROM system:indexes WHERE state <> 'ready'; against the query service.
If the stuck index appears, allow the indexer to finish. On the indexer node, confirm active builds drop to zero and the index no longer appears in non-ready states.
Poll the query at intervals until it returns empty and remains empty across two consecutive checks before resuming normal traffic.
Drop and recreate only indexes that are stuck, corrupt, or storage-incompatible and cannot complete the build. Do not rebuild all indexes preemptively.

Missing diagnostic detail

What Couchbase Server version and index storage type are you running? Exact indexer log strings, admin REST paths, and progress fields vary by release, so the version determines whether scripting against log completion messages is safe.

Verification

SELECT name, keyspace_id, state FROM system:indexes WHERE state <> 'ready';

An empty result plus zero active indexer builds confirms readiness. Check indexer node logs for a build-completed message for the affected index definition; exact log text is version-sensitive and should be verified against your release.

Assumptions and uncertainty

Assumes Couchbase Server 6.x or 7.x with standard N1QL Global Secondary Indexes.
Standard GSI, array, partial, and covering indexes generally persist across upgrade; manual rebuild is typically reserved for stuck or storage-incompatible indexes.
system:indexes state can lag briefly after the indexer reports completion; do not rely on a single check.

Couchbase Upgrade: Indexes Stuck in 'Indexing' State During Upgrade

1 answer

Direct answer

Confirmed indicators versus uncertain signals

Steps for this case

Missing diagnostic detail

Verification

Assumptions and uncertainty

Your perspective belongs here.

0 question comments