CouchDB backup restoration: decide between _replicate endpoint and direct file copy for large databases

Answer

For a large CouchDB database, _replicate is the safer choice when the source must stay online. It streams a consistent view, preserves compaction state, attachment store format, and verifies checksums. A direct file copy can be used only if you can stop the source or take an OS‑level snapshot to avoid data races.

Confirmed Facts

• _replicate streams the full database view, including compaction state and attachment store format.
• Direct file copy copies raw _files and metadata but does not lock the database; concurrent writes can lead to inconsistent state.
• Checksum verification is built into the replication protocol; a file copy requires post‑copy checks.
• Stopping CouchDB or taking a snapshot guarantees a quiescent view for a file copy.

Likely Explanation

Because CouchDB does not lock its on‑disk files during writes, the file system view can change mid‑copy. Replication bypasses this by serializing changes and ensuring the target database reaches the same sequence number as the source.

Recommended Steps

Using _replicate

1. Ensure the target database does not exist or is empty.
2. POST to http://source:5984/_replicate:

   {
     "source": "source_db",
     "target": "http://target:5984/target_db",
     "create_target": true
   }

3. Monitor progress via the progress field or by querying _active_tasks.
4. After completion, run couchdb-check target_db to verify document and attachment integrity.
5. Optionally, compare _db_info and _metadata from source and target to confirm compaction state.

Direct File Copy (when quiescent)

1. Stop CouchDB on the source node (or freeze the database via a snapshot).
2. Copy the database directory:

   rsync -a /var/lib/couchdb/source_db/ /var/lib/couchdb/target_db/
   

3. Restart CouchDB on the target and run couchdb-check target_db.
4. Compare _db_info and _metadata to ensure compaction state matches.

When to Use Direct Copy

Only if you can guarantee the source is not writing during the copy:

• Scheduled maintenance window.
• OS snapshot or filesystem‑level snapshot (e.g., LVM, ZFS).

Verification Checklist

• couchdb-check on the target.
• Sequence numbers: _db_info update_seq matches source.
• Attachment checksums: compare _attachments digests.
• Compaction state: _metadata compact flag.

Missing Diagnostic Detail

Do you have the ability to stop the source CouchDB instance or take an OS‑level snapshot before copying?