CouchDB Upgrade Utility: Recovering from Mixed-State Databases After an Interrupted 3.x to 4.x Transition

Short answer

When the _upgrader stops in the middle of a 3.x‑to‑4.x transition, CouchDB does not provide an automatic resume or checkpoint. The only supported recovery path is a full restore from a known‑good backup. If no backup exists, the safest option is to wipe the data directory and rebuild the cluster from scratch or from replication sources, because manual file‑level cleanup carries a high risk of permanent data loss.

Why the upgrade utility can’t resume

The _upgrader performs an in‑place rewrite of each .couch file. It does not keep a per‑file or global migration state. Consequently:

• There is no atomic transaction that guarantees all files are upgraded together.
• When the process is killed, some files may already contain the new 4.x B‑tree layout while others remain in the old format.
• The utility has no built‑in idempotency check; a second run on a mixed‑state directory can corrupt files that were already upgraded.
• Release notes for 4.x explicitly state that the upgrader “does not support resuming” and that operators should restore from a backup if an interruption occurs.

Detecting a partial migration

There is no automated flag, but you can identify the problem by inspecting CouchDB logs and the file metadata:

1. Start CouchDB in a paused state (e.g., stop the service and run couchdb -vv in debug mode) to avoid automatic startup.
2. Check couchdb.log for errors such as bad file format or invalid version that reference specific .couch files.
3. Use couchdb -vv or couchdb -vvv to print the version stored in the data directory’s couchdb.dir metadata. If the binary version (4.x) and the directory version differ, a partial upgrade is likely.
4. Run a quick file scan: find /var/lib/couchdb -name "*.couch" -exec stat -c "%n %y" {} \; | grep -i upgrade to spot files that were modified during the last upgrade run.

Recommended recovery path when no backup is available

Because manual repair can corrupt data, the following steps are the safest:

1. Stop CouchDB. Ensure no nodes are running to avoid further writes.
2. Back up the current state. Even if it’s corrupted, copy the entire /var/lib/couchdb to a safe location. This preserves the data for forensic analysis or for incremental attempts.
3. Delete or rename the data directory. CouchDB will create a fresh directory on startup.
4. Re‑initialize the cluster. Re‑create the cluster configuration (e.g., couchdb -C config.ini) and let the nodes bootstrap.
5. Restore data from replication. If you have a healthy 3.x source node or a recent snapshot, use _replicate or couchdb‑backup tools to bring the data back into the new 4.x cluster.
6. Validate configuration files. The upgrader does not touch local.ini or default.ini. After the fresh install, manually copy any custom settings from the old local.ini into the new one, and restart the node to verify they are applied.

When a backup exists

Restore the backup exactly as it existed before the upgrade attempt. Verify the data directory’s couchdb.dir metadata shows the pre‑upgrade version. Then run the _upgrader again from a clean state; the process will now upgrade all files atomically.

Key take‑aways

• The _upgrader does not support resuming; a mixed‑state directory is considered unrecoverable without a backup.
• Release notes for 4.x explicitly advise restoring from a backup after an interruption.
• Configuration files are not migrated automatically; they must be updated manually after a fresh install.
• If you lack a backup, the safest path is to delete the corrupted data directory and rebuild the cluster from a clean source or via replication.