Diagnosing Stalled Continuous Replication in CouchDB 3.x

Recognizable condition

A continuous replication job appears to be stuck: the replication document shows "state": "triggered" but the fields docs_read, docs_written or replication_id_version do not increase after several minutes.

Cause / diagnostic table

Symptom in logs / replication doc	Likely cause
HTTP 401/403 errors	Authentication or authorization misconfiguration
ECONNRESET, timeout, or connection reset by peer	Network partition or intermittent connectivity
No error, but docs_read stalls while source DB shows high pending changes	Design document view indexing lag
Large replication payloads, frequent retries, no progress	Excessive revision tree depth from unresolved conflicts

Ordered checks

1. Inspect the replication document

   GET /_replicator/
   

   Look at state, progress (docs_read, docs_written). If unchanged for >5 min, proceed.
2. Check CouchDB logs for replication entries

   grep -i replication /opt/couchdb/log/couch.log | tail -n 20
   

   Search for patterns: 401 Unauthorized, 403 Forbidden, ECONNRESET, timeout.
3. Verify network connectivity between source and target

   curl -v http://:5984/ 2>&1 | grep -E "HTTP|SSL|Connected"
   

   Ensure TLS handshake succeeds (if using HTTPS) and no premature connection closure.
4. Assess source database view indexing lag

   GET //_active_tasks
   

   Look for tasks with type": "indexer" and high progress or long elapsed_time.
5. Check revision tree depth / conflict count

   GET //_local/documents?conflicts=true
   

   If many documents show _conflicts arrays, the revision tree may be bloated.

Fixes tied to findings

• Authentication/authorization issue
  • Verify that the replication document’s source or target URL includes correct credentials (e.g., http://user:pass@host:5984/db) or that a matching _users doc exists.
  • Ensure the _replicator database has appropriate _security settings allowing the replicator user to read/write.
• Network partition
  • Check firewall rules, load balancer timeouts, or VPN stability between nodes.
  • If using TLS, confirm certificates are valid and not expired.
  • Consider setting "retries_per_request": 5 and "socket_options": { "keepAlive": true } in the replication document to improve resilience.
• View indexing lag
  • Trigger a view update manually: POST //_design//_view/?stamp=now.
  • If indexing is chronically slow, review the view’s map function for inefficiencies or add missing indexes.
• Excessive revision tree depth
  • Run a compaction on the source database: POST //_compact.
  • After compaction, monitor the replication document; docs_read should begin increasing.
  • If conflicts are the root cause, inspect conflicting revisions (GET //?conflicts=true) and resolve them manually before allowing automatic conflict resolution.

Escalation criteria

• Replication remains stuck after completing all checks and applying the corresponding fix.
• Logs show repeated out_of_memory or heap exhaustion messages when increasing batch size.
• Network tests reveal persistent packet loss or latency >200 ms that cannot be resolved locally.

In these cases, gather the replication document, relevant log excerpts (grep replication), and output of curl -v and _active_tasks, then open a support ticket with the CouchDB team or consult the project’s mailing list.

Practical verification

After applying a fix, re‑run the first check: GET /_replicator/. A successful sign is that state changes to completed (for a one‑shot replication) or that docs_read and docs_written increase steadily over a 2‑minute window.