Making CouchDB Replication Truly Incremental: Using Since and Until

Problem: Avoiding Unnecessary Data Transfer Over Unreliable Links

When you run CouchDB on edge devices or in a mesh network, you often need to synchronize only the changes that have happened since the last successful sync. Sending the whole database each time wastes bandwidth, drains batteries, and can stall replication on flaky connections.

Thesis: Leverage the since and until parameters of the _replicate endpoint to create true incremental sync windows.

How Incremental Replication Works Under the Hood

CouchDB assigns every document a monotonically increasing update sequence number (the value you see in the _local replication document’s seq field). When you start a replication job you can tell CouchDB to begin at a specific sequence number (since) and stop at another (until). The replicator then walks the revision tree, sending only those leaf revisions whose sequence falls inside the window. Unchanged documents are never re‑serialized, and conflict detection still works because each revision carries its full history.

Worked Example: One‑Shot Incremental Pull Between Two Local Nodes

Assume two CouchDB instances running locally:

• Source: http://localhost:5984 (db: mydb)
• Target: http://localhost:5985 (db: mydb)

1. Create a document on the source:

curl -X PUT http://localhost:5984/mydb/foo \
  -H "Content-Type: application/json" \
  -d '{"title":"first version"}'

2. Replicate the whole database once to bootstrap the target (this creates the initial checkpoint):

curl -X POST http://localhost:5985/_replicate \
  -H "Content-Type: application/json" \
  -d '{"source":"http://localhost:5984/mydb","target":"mydb","create_target":true}'

3. Check the replication state on the target to learn the last processed sequence:

curl http://localhost:5985/mydb/_local/replication-<uuid>

The response contains a field like "seq":"1234". Save that value.

4. Now make a change on the source (a new revision of the same doc):

curl -X PUT http://localhost:5984/mydb/foo \
  -H "Content-Type: application/json" \
  -d '{"title":"second version","_rev":"1‑abc"}'

5. Run an incremental pull, telling the replicator to start after the saved sequence:

curl -X POST http://localhost:5985/_replicate \
  -H "Content-Type: application/json" \
  -d '{"source":"http://localhost:5984/mydb","target":"mydb","since":"1234"}'

6. Inspect the target document; you should see the new revision and no re‑transfer of the unchanged parts:

curl http://localhost:5985/mydb/foo

Conflict Awareness During Incremental Sync

If the same document ID is updated on both sides between two sync points, CouchDB will store both revisions and mark the document with a _conflicts array. The replicator does not resolve conflicts automatically; it merely propagates them. To keep storage bounded, implement a conflict‑resolution policy (e.g., application‑side merge or a “winner‑wins” script) and run it after each incremental session.

Trade‑offs and Limitations

• Checkpoint durability: If the replication process crashes before it updates its _local state document, you may replay the same changes or, worse, miss a window if you rely solely on the stored since value. Mitigate by persisting the last sequence yourself (e.g., in a file or a dedicated config doc) and using it for the next job.
• Stale reads with stale=update_after: Some clients use this mode to get fast reads while letting updates happen in the background. After an interrupted replication, the view index may lag behind the actual sequence, so you must verify consistency on the client side if strong guarantees are required.
• Bandwidth savings vs. latency: Very small windows reduce data transferred but increase the number of HTTP round‑trips. Choose a window size that matches your network characteristics (e.g., a few hundred changes per batch on a high‑latency link).

Actionable Closing

1. Bootstrap the target with a full replication to establish an initial checkpoint.
2. After each successful sync, record the seq from the replication’s _local document.
3. Use that value as the since parameter for the next incremental job.
4. Monitor the _conflicts field on documents; schedule a conflict‑resolution routine if the array grows beyond a threshold.
5. Test the flow by interrupting a replication (e.g., stop the source mid‑sync) and confirming that the next run resumes from the saved sequence without missing or duplicating changes.

By treating the replication sequence number as a durable cursor, you turn CouchDB’s MVCC‑powered replication into a bandwidth‑efficient, offline‑first primitive suitable for unreliable networks.