Building Incremental Sync with CouchDB's _changes Feed

The problem: syncing without scanning everything

You have a CouchDB database that receives writes from multiple sources—mobile apps, edge devices, backend services. You need to propagate those writes to another database, a search index, or a downstream analytics pipeline. Polling the entire database on every run is wasteful; you only want what changed since the last time you checked. CouchDB's _changes feed solves exactly this: it exposes a persistent, append‑only log of every revision with a monotonically increasing sequence number, so you can fetch incremental updates without a full scan.

Feed semantics and the parameters you actually use

Each database maintains its own _changes feed. A GET request to /db/_changes returns a JSON object with a results array and a last_seq value. Each entry in results contains the document _id, the current rev, and a changes array listing the revision history for that update. The feed is ordered by the per‑database sequence number, not by wall‑clock time, so out‑of‑order writes appear in the order they were committed to the database.

The parameters that matter in practice:

• since – the sequence number to start from (use the last_seq from your previous response).
• limit – cap the number of rows returned per request.
• feed – normal (one‑shot), longpoll (blocks until changes arrive), or continuous (keeps the HTTP connection open, sending newline‑delimited JSON).
• heartbeat – milliseconds between keep‑alive newlines in continuous mode; helps detect dead connections.
• include_docs=true – attach the full document body to each change, avoiding a second fetch.

Filtered replication: same feed, narrower scope

When you only need a subset of documents—say, per‑tenant data or documents tagged for a specific region—you can attach a filter function in a design document. The filter runs on the source node during replication or when you query _changes?filter=ddoc/filterName. It receives the document and the request parameters; return true to include the change. This is the mechanism CouchDB's replicator uses for filtered replication, and it's equally available for custom consumers.

Example design document with a filter that only passes documents where tenant_id matches a query parameter:

{
  "_id": "_design/sync",
  "filters": {
    "by_tenant": "function(doc, req) { return doc.tenant_id === req.query.tenant_id; }"
  }
}

Then request /db/_changes?filter=sync/by_tenant&tenant_id=acme-corp&since=12345. The filter executes on the server, so only matching changes travel over the wire—critical for bandwidth‑constrained edge links.

Worked example: incremental polling loop

The pattern below runs on any host that can reach CouchDB (application server, CI job, edge gateway). It assumes you have a persistent store (file, Redis, another CouchDB doc) to save the last processed last_seq.

#!/usr/bin/env bash
# poll_changes.sh – run periodically via cron, systemd timer, or a long‑lived process

COUCH_URL="http://admin:secret@localhost:5984/mydb"  # replace with real credentials
STATE_FILE="/var/lib/myapp/last_seq.txt"

# Read last sequence or start from 0
LAST_SEQ=$(cat "$STATE_FILE" 2>/dev/null || echo 0)

# Fetch up to 500 changes since last sequence
RESPONSE=$(curl -s -G "${COUCH_URL}/_changes" \
  --data-urlencode "since=${LAST_SEQ}" \
  --data-urlencode "limit=500" \
  --data-urlencode "include_docs=true" \
  --data-urlencode "feed=normal")

# Extract new last_seq (jq required)
NEW_SEQ=$(echo "$RESPONSE" | jq -r '.last_seq')

# Process each change – replace with your actual logic
# Example: print doc id and revision
echo "$RESPONSE" | jq -c '.results[]' | while read -r row; do
  DOC_ID=$(echo "$row" | jq -r '.id')
  REV=$(echo "$row" | jq -r '.changes[0].rev')
  echo "Processing $DOC_ID @ $REV"
  # Your downstream write, index update, etc. here
  # If you need the full doc: DOC=$(echo "$row" | jq -c '.doc')
done

# Persist new sequence for next run
echo "$NEW_SEQ" > "$STATE_FILE"

Where to run: any machine with network access to CouchDB and curl + jq installed. Permissions: the CouchDB user needs _reader role on the source database. Risks: if the script crashes after processing but before writing STATE_FILE, you'll reprocess the same changes on the next run—design your downstream to be idempotent (use _id + _rev as a deduplication key). Verification: after a run, confirm STATE_FILE matches the last_seq in the response and that no duplicate _id/_rev pairs appear in your downstream.

Trade‑offs and limitations you'll hit

• Conflicts are not resolved. When a document has multiple leaf revisions, each appears as a separate change entry. Your consumer must detect conflicts (multiple entries for the same _id with different rev values) and apply your resolution policy—last‑writer‑wins, application‑specific merge, or manual intervention.
• Old since values replay everything. Compaction removes old document bodies but does not truncate the changes feed history. If you restart from since=0 on a large, active database, you'll read millions of rows. Always persist last_seq and never reset it unless you truly need a full replay.
• Continuous feed holds connections. feed=continuous keeps an HTTP connection open indefinitely. Heartbeats prevent silent drops, but each consumer consumes a file descriptor on the CouchDB node. For many consumers, prefer longpoll with a reasonable timeout or a dedicated replication job.
• Filter functions add CPU. Filters run on the source node for every change. Complex logic (regex, external lookups) increases replication latency and can become a bottleneck. Keep filters pure, deterministic, and fast.
• Database‑scoped, not cluster‑scoped. If your application writes to multiple databases, you must aggregate feeds yourself—there is no single cluster‑wide changes stream.

What to verify before you ship

1. Start a local CouchDB (docker run -p 5984:5984 couchdb:3.4), create a test database, write three documents, then request /_changes?since=0. Confirm you see seq, id, and changes fields.
2. Add the design document filter above, write documents with and without tenant_id, then query /_changes?filter=sync/by_tenant&tenant_id=acme-corp. Only matching documents should appear.
3. Run the polling script twice in a row. The second run should return an empty results array and the same last_seq.
4. Trigger compaction (POST /mydb/_compact), then poll from a recent since. The feed should still return changes; older history is simply no longer needed for normal operation.

Closing: treat the feed as a log, not a queue

The _changes feed gives you a reliable, ordered log of every mutation. Build your consumers to be idempotent, persist last_seq durably, and handle conflicts explicitly. For most eventually‑consistent architectures—mobile sync, search indexing, audit trails, cross‑region replication—this primitive is simpler and more robust than building custom triggers or CDC pipelines. Start with the polling script above, add your business logic, and you have a production‑grade incremental sync foundation.