Diagnosing Missing or Truncated Results from web3.js getPastEvents

You call contract.getPastEvents('Transfer', { fromBlock: 0, toBlock: 'latest' }) to rebuild a token's history, and one of three things happens: the call hangs until the provider times out, it throws a response-size or rate-limit error, or it returns an array that is silently shorter than the real event count. The root cause is almost always the same: getPastEvents is a thin wrapper over the eth_getLogs JSON-RPC method, and every node provider caps how much work a single log query may do. The fix is to stop asking for the whole chain in one request and page through bounded block ranges instead.

This guide assumes web3.js 1.x (the contract.getPastEvents(event, options) signature) against an HTTP provider such as Infura, Alchemy, or a local Hardhat node. Version 4.x keeps the same underlying RPC behavior, so the diagnostics apply there too.

Recognizing the failure mode

Before changing code, match your symptom to a cause. The table below covers the conditions seen most often when paging event history.

Ordered checks before rewriting anything

1. Confirm the events exist at the RPC layer. Bypass web3.js entirely. From any machine with network access to your endpoint (no special permissions needed), run:

   curl -X POST $RPC_URL \
     -H "Content-Type: application/json" \
     -d '{
       "jsonrpc": "2.0",
       "id": 1,
       "method": "eth_getLogs",
       "params": [{
         "address": "0xYOUR_CONTRACT",
         "fromBlock": "0x0",
         "toBlock": "0x64"
       }]
     }'

   Replace $RPC_URL and the contract address. If this returns logs for a small range, your filter and address are fine and the problem is range width or paging logic. If it returns nothing even for a range where you emitted events on a testnet, suspect the address or event signature.
2. Measure event density. Query a 100-block sample and count results. If 100 blocks yield 500 logs, a 100,000-block range would demand roughly 500,000 logs in one response — far beyond any provider cap. This number determines your page size.
3. Check your boundary arithmetic. Both fromBlock and toBlock are inclusive. A loop that sets the next fromBlock equal to the previous toBlock re-fetches that block and produces duplicates; a loop that skips a block loses events. The next page must start at previousToBlock + 1.
4. Rule out rate limiting. If failures appear only after several successful pages, add a log line recording the HTTP status of each request. HTTP 429 or provider-specific rate errors confirm throttling, not a logic bug.

Fix: paginate with bounded, non-overlapping ranges

The reliable pattern is a loop that walks forward in fixed-width windows, with retry and backoff per window. Run this wherever your sync job runs (Node.js process, no elevated permissions required beyond network access):

const PAGE_SIZE = 2000; // blocks per request; tune down if you still hit caps

async function fetchAllEvents(contract, eventName, startBlock, endBlock) {
  const results = [];
  for (let from = startBlock; from <= endBlock; from += PAGE_SIZE) {
    const to = Math.min(from + PAGE_SIZE - 1, endBlock); // inclusive, no overlap
    const chunk = await withRetry(() =>
      contract.getPastEvents(eventName, { fromBlock: from, toBlock: to })
    );
    results.push(...chunk);
  }
  return results;
}

async function withRetry(fn, attempts = 5) {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === attempts - 1) throw err;
      const delay = 500 * 2 ** i; // exponential backoff: 0.5s, 1s, 2s...
      await new Promise(r => setTimeout(r, delay));
    }
  }
}

Two details matter here. First, from + PAGE_SIZE - 1 combined with the next page starting at from + PAGE_SIZE guarantees inclusive ranges with zero overlap and zero gaps. Second, the backoff handles transient 429s and connection resets without aborting a multi-hour sync. Persist the last completed toBlock to a file or database after each page so a crash lets you resume instead of restarting.

Fix: decode and validate what comes back

web3.js decodes returnValues for you when the contract ABI includes the event, but if you ever drop to raw eth_getLogs (for example, to query multiple contracts at once), you receive ABI-encoded topics and data hex strings that you must decode against the ABI yourself. A practical sanity check either way: every returned entry should have a blockNumber inside the requested range and a monotonically non-decreasing sequence of (blockNumber, logIndex) pairs. If a page contains a block number outside its range, your provider is misbehaving — treat the page as untrusted and re-fetch it.

Verify the fix on a local testnet

Do not validate pagination logic against mainnet first. On a local Hardhat or Ganache node, deploy a contract, emit a known event once per block across 200 blocks, then run your paging function with PAGE_SIZE deliberately small (say 25) to force multiple pages. Assert that you receive exactly 200 events, that block numbers 0–199 each appear exactly once, and that no duplicates exist. Then run the raw eth_getLogs curl command above over the full range and confirm the counts match. This catches off-by-one boundary bugs deterministically, which mainnet testing cannot.

Limitations and when to escalate

Pagination solves provider caps but not fundamental scale problems. Escalate to a different architecture when you hit any of these:

• Sync time is unacceptable. Walking millions of blocks sequentially takes hours even with perfect paging. An indexing service (The Graph, or your own database fed by a one-time backfill plus a live subscription) is the standard answer.
• You need real-time data. getPastEvents is historical only; use contract.events.Transfer() subscriptions for new events, and keep the paged backfill for catching up after downtime.
• Reorg safety matters. Near the chain head, recently fetched blocks can be reorganized. Stop pagination a safe margin (commonly 12+ blocks on Ethereum mainnet) behind the head and re-fetch that tail region on the next run.
• The provider's archive access is inconsistent. Some providers route old-block queries to archive nodes with different limits. If identical ranges intermittently fail, test the same query against a second provider before blaming your code.

The durable takeaway: treat getPastEvents as a bounded-range primitive, never a full-history query. Size pages from measured event density, keep ranges inclusive and non-overlapping, retry with backoff, and persist progress so any failure is a resume, not a restart.