CouchDB as a Sync Engine: Replication First, Queries Second

An app that has to keep working on a train, in a basement, or behind a captive portal has one hard requirement: reads and writes happen locally, and reconciliation happens later. CouchDB ships a documented, resumable, bidirectional HTTP replication protocol aimed at exactly that case. The practical engineering decision is whether you build your sync layer on top of it or beside it.

The thesis here: treat replication and the _changes feed as your product's sync API, and treat Mango (_find) as a server-side convenience for reporting and admin screens. Teams that invert this — polling a view or _all_docs and diffing on the client — reimplement replication badly, and inherit its hard cases (deletes, conflicts, resumability) without its guarantees.

The pull side: _changes is a cursor, not a table scan

GET /{db}/_changes?since={seq} returns the documents that changed after a sequence token. Useful parameters include limit to page, heartbeat to keep a long poll alive, style=all_docs to include conflicting leaf revisions, and filter to narrow the stream.

The sequence token is opaque. Persist it verbatim; do not parse it, order it numerically, or assume its internal format is a stable contract. It is the only thing standing between a resumed sync and a full rescan.

Run this from a shell against your server. CouchDB 3.x requires an admin credential for most database operations, so substitute a real user and password:

curl -u admin:$COUCH_PASS "$COUCH_HOST/myapp/_changes?since=0&limit=5"

Expect a JSON body containing a results array and a last_seq. Store last_seq, then re-request with since set to that value. The check that matters: the second response should contain only changes newer than the first batch, with no repeats.

The push side: one-shot _replicate or scheduled _replicator

Replication is incremental and idempotent. If a transfer is interrupted, re-running it resumes from the last checkpoint rather than starting over. You can trigger it once with POST /_replicate, or declare it as a document in the _replicator database, which the server schedules and retries on your behalf. Persistent replication documents are usually the better fit for a device or a regional node that should stay in sync without your application orchestrating every round.

A minimal _replicator document, created with PUT to /_replicator/{id}:

{
  "source": "http://syncuser:$COUCH_PASS@source-host:5984/myapp",
  "target": "http://syncuser:$COUCH_PASS@target-host:5984/myapp",
  "continuous": true
}

Note the credential embedded in the URL. Replication documents store it, so restrict who can read _replicator and prefer a dedicated user over an admin account. Expected check: write a document on the source, then read it on the target. Interrupt the link mid-transfer and confirm the target catches up without duplicating revisions.

Conflicts are your policy, not a database setting

CouchDB does not reject concurrent writes to the same document id. It keeps every leaf revision in a revision tree and picks a deterministic winner (higher generation, then higher revision hash). The losing revisions stay on disk until something removes them.

That makes conflict resolution application policy. If you never define a merge rule and never delete losing revisions, conflicts accumulate silently, and your "current" document is whatever the deterministic rule selected — which may not be the edit the user made last.

To see the leaves, request the document with conflicts expanded:

curl -u admin:$COUCH_PASS "$COUCH_HOST/myapp/mydoc?conflicts=true"

Expect a _conflicts array when two replicas edited the same id independently. Your sync code should read those leaves, apply a rule (last-writer-wins with a trusted clock, field-level merge, or a user prompt), write the merged revision, and delete the losers explicitly.

Where Mango fits — and where it doesn't

Mango is the supported way to run structured queries: declare an index with _index, query with _find. Without a matching index the planner falls back to a full scan, so check the explain option when a query feels slow. Indexes are eventually consistent with primary data, which surprises people in sync designs: a query issued immediately after a write may not see that write yet. Do not use a query result as proof that a replication has landed.

Two limits worth stating plainly. CouchDB is not relational — no joins, no multi-document transactions, no ad-hoc aggregation. And deletes are tombstones: a deleted document keeps its id and revision history so the delete can replicate. Compaction and purge are separate administrative operations, and purging removes revision history in a way that can strand replicas that have not yet caught up.

Partitioned databases, if your sync is per-tenant

CouchDB 3.x partitioned databases co-locate documents that share a partition key, letting _all_docs and _changes be scoped to a single partition. For per-user or per-tenant sync this is a good fit, but it constrains document ids and query patterns up front, so decide before you have data to migrate.

Verify before you commit

1. GET / on the target server and read the exact version string. Partitioned databases, selector-based filtered replication, and first-run admin setup are 3.x behaviors; 2.x and 1.x differ.
2. Exercise the pull side manually with _changes?since=0&limit=5, then page forward using last_seq.
3. Create a _replicator document between two local databases, write on the source, and confirm arrival on the target. Restart mid-transfer to observe checkpoint resumption.
4. Force a conflict on purpose, inspect it with ?conflicts=true, and confirm your resolution code removes the losing leaf.
5. Compare a _find query with and without a declared index using explain.

If you take one thing away: the sequence token and the revision tree are the two pieces of state your sync layer must own. Everything else — indexes, filters, partitioning — is tuning around them.