web3.js Event Subscriptions: Chunked Log Ranges and Reorg-Safe Checkpoints

If you have ever built an EVM indexer with web3.js, you have probably hit the same wall: an open-ended event subscription looks fine until the provider drops the websocket, or a chain reorganization silently rewrites history. The practical decision is to stop relying on a single long-lived subscription and instead poll bounded block ranges, advancing a checkpoint only after those blocks reach a safe confirmation depth.

Why Open-Ended Subscriptions Break

When you call contract.events.EventName({ fromBlock: 'latest' }), you are asking the node to push matching logs as new blocks arrive. That works until it doesn't. Many hosted JSON-RPC providers cap the block range accepted by eth_getLogs, terminate idle websocket connections, or experience lag during high load. If your process reconnects and resumes from latest, any events emitted in the gap are lost.

A chain reorganization (reorg) makes the problem worse. A reorg happens when the network replaces recently mined blocks with a competing chain. Events you already stored may belong to orphaned blocks. If you treat every incoming log as permanent, your index drifts from the canonical chain.

The Mechanism: Bounded Ranges and Confirmations

The safer pattern is to treat eth_getLogs as a paginated query. You define a chunk size—for example, 2,000 blocks—and walk forward from a persisted checkpoint. You only advance the checkpoint after the chunk is stored and the latest block in that chunk is older than a confirmation threshold (often 12 or more blocks on Ethereum mainnet, though the right number depends on the chain and your risk tolerance).

In web3.js, the contract wrapper handles topic encoding for you. The following pattern illustrates the shape of a resumable indexer; it is a starting template, not a tested production script.

const { Web3 } = require('web3');
const web3 = new Web3('wss://your-provider.example/v1/YOUR_KEY');

const contract = new web3.eth.Contract(abi, contractAddress);

const CHUNK_SIZE = 2000;          // tune to your provider's eth_getLogs limit
const CONFIRMATIONS = 12;         // blocks to wait before finalizing

async function indexFrom(checkpoint) {
  const latest = await web3.eth.getBlockNumber();
  const safeHead = latest - CONFIRMATIONS;
  if (safeHead <= checkpoint) return;

  const end = Math.min(checkpoint + CHUNK_SIZE, safeHead);

  // Explicit window avoids provider range caps and idle-subscription drops
  const events = await contract.getPastEvents('Transfer', {
    fromBlock: checkpoint + 1,
    toBlock: end
  });

  for (const evt of events) {
    if (evt.removed) {
      // Reorg delete: remove this event from your store
      await deleteEvent(evt.transactionHash, evt.logIndex);
    } else {
      await saveEvent({
        txHash: evt.transactionHash,
        logIndex: evt.logIndex,
        blockNumber: evt.blockNumber,
        args: evt.returnValues
      });
    }
  }

  // Persist checkpoint only after the chunk is durable
  await saveCheckpoint(end);
}

Two details matter here. First, fromBlock and toBlock are explicit integers, not 'latest'. Second, the checkpoint moves only after the loop finishes. If the process crashes mid-chunk, you resume from the last durable checkpoint and reprocess the same range. Because you store transactionHash and logIndex, reprocessing is idempotent: you can upsert rather than blindly append.

Raw Logs and Topic Filtering

If you need logs from multiple contracts or do not want to instantiate a contract object, subscribe to raw logs instead. Build the topic array with web3.eth.abi.encodeEventSignature(abiItem) and decode payloads with web3.eth.abi.decodeParameters(inputs, data). Filter server-side by address and indexed topics; fetching every log and filtering in application code wastes bandwidth and RPC quota.

Version and Provider Assumptions

web3.js has undergone major API changes between the v1.x line and v4. Package layout, subscription options, and the exact shape of contract event objects can differ. Before you rely on a field such as removed, verify that your installed version exposes it. Pin the dependency in your lockfile and run the same code against your target provider in staging.

Provider behavior is equally variable. Some nodes enforce strict eth_getLogs range limits and return errors for large windows. Websocket stability varies by vendor. Treat your provider as an untrusted boundary: add retries with exponential backoff, heartbeat checks, and a backfill path that can replay from an older checkpoint if a subscription gap is detected.

Common Mistakes

• Using latest as a starting point. It is a moving target. A node under load may report a block that is not yet propagated, or you may miss logs during a brief disconnect.
• Oversized chunks. A 500,000-block range will likely be rejected or time out. Probe your provider by increasing the range until it errors, then set your chunk size comfortably below that limit.
• Ignoring reorg deletes. If your store only inserts, orphaned events remain forever. Check the removed flag (when available) and reconcile affected ranges.
• Mismatched ABIs. Decoding with the wrong event signature produces silently wrong values. Confirm topic order and parameter types against the contract source.
• Unbounded listeners. Attach .on('data'), .on('error'), and .on('close') handlers, and remove subscriptions when a worker shuts down.

How to Check the Result

You do not need mainnet to validate the logic. Deploy a contract that emits a known event to a local development chain, start indexing from the deployment block, and assert that decoded arguments match expectations after the confirmation threshold passes. Then force a reorg on that local chain (or a fork), emit new events, and confirm that removed logs are purged and the backfill path converges to the same final state.

For provider limits, run a probe script that calls eth_getLogs with progressively larger ranges and record where errors begin. Finally, compare your event count against an independent archive node or block explorer for the same block range. If counts diverge, your chunk boundaries or confirmation depth are likely misaligned.

Event indexing is not hard because the code is complex; it is hard because the chain is eventually consistent and the provider is imperfect. Bounded ranges, durable checkpoints, and explicit reorg handling turn a fragile subscription into a resumable job.