Paginate historical Ethereum events with web3.js

The problem: one big event query is fragile

You need every Transfer event for an ERC-20 token from its deployment block to the present. A single call to web3.eth.getLogs with fromBlock: 0 and toBlock: 'latest' looks simple, but Ethereum nodes must scan every block in that range, match topics, and return the full result set. Public RPC providers often reject or time out wide ranges, and even a private node can run out of memory. Because eth_getLogs has no cursor or offset, you cannot simply request page two; you must manage the block range yourself.

The practical takeaway is to split the historical range into fixed-size chunks, query each chunk with explicit fromBlock and toBlock bounds, merge the results, and pause between requests so you stay within RPC rate limits.

Worked example: chunked ERC-20 Transfer logs

This Node.js example assumes web3.js v4.x. Replace the placeholders with your RPC endpoint and contract address. It runs in a standard Node process; no special filesystem permissions are required beyond network access to the RPC.

import { Web3 } from 'web3';

const RPC_URL = '';
const CONTRACT_ADDRESS = '';
const FROM_BLOCK = 0n; // or a specific deployment block
const TO_BLOCK = 'latest';
const CHUNK_SIZE = 1000n; // blocks per RPC call

const web3 = new Web3(RPC_URL);

// Transfer(address indexed from, address indexed to, uint256 value)
const TRANSFER_TOPIC = web3.utils.sha3('Transfer(address,address,uint256)');

async function fetchLogsInChunks(fromBlock, toBlock, chunkSize) {
  let current = fromBlock;
  const allLogs = [];

  while (current <= toBlock) {
    const end = current + chunkSize - 1n;
    const boundedEnd = end > toBlock ? toBlock : end;

    const filter = {
      address: CONTRACT_ADDRESS,
      fromBlock: current,
      toBlock: boundedEnd,
      topics: [TRANSFER_TOPIC],
    };

    const logs = await web3.eth.getLogs(filter);
    allLogs.push(...logs);

    console.log(`Retrieved ${logs.length} logs for blocks ${current} to ${boundedEnd}`);
    current = boundedEnd + 1n;

    // Throttle to reduce 429 risk on public RPCs.
    await new Promise((resolve) => setTimeout(resolve, 200));
  }

  return allLogs;
}

// Usage
fetchLogsInChunks(FROM_BLOCK, TO_BLOCK, CHUNK_SIZE)
  .then((logs) => {
    console.log(`Total Transfer logs: ${logs.length}`);
  })
  .catch((error) => {
    console.error('Pagination failed:', error.message);
  });

Expected check: each log object includes blockNumber, transactionHash, and topics. The first topic should match TRANSFER_TOPIC, and every blockNumber should fall inside the requested chunk. If the array is empty for a range you know contains transfers, confirm the contract address and topic encoding.

Risk: choosing CHUNK_SIZE too large can trigger provider timeouts or maximum response-size errors. Choosing it too small multiplies request count and increases the chance of hitting rate limits. Start with 1,000 blocks and adjust based on your provider's errors.

Rate limits, retries, and filter state

Public RPC endpoints commonly return HTTP 429 when you send too many requests in a short window. The example above adds a fixed 200 ms pause, but a more robust approach is to catch 429 errors, wait with exponential backoff, and retry the same chunk. Do not increase chunk size to bypass rate limits; that shifts the problem from request count to response size.

If you use web3.js filter objects instead of getLogs, remember that filters hold provider state. Call filter.uninstall() when you are done so the node can release the subscription. For one-off historical retrieval, getLogs is stateless and usually the simpler choice.

Limitations and how to verify the result

• Recent blocks can be reorganized. Treat the most recent 10–100 blocks as tentative and re-query them later if your application requires finality.
• Providers often enforce a maximum block range per eth_getLogs call. If a chunk fails, reduce CHUNK_SIZE.
• Anonymous events do not use topics[0]; filter by address only for those contracts.
• This pattern retrieves historical logs only. It does not replace real-time subscriptions for new events.

Practical verification: run a single eth_getLogs request over the full range using curl or your provider's developer console, then compare the total count against the sum of your chunks. You can also check a block explorer such as Etherscan for the same contract and block range, though explorer counts may exclude internal transactions or differ by finality.

Actionable closing

Start with a 1,000-block chunk size, log the block range for every request, and retry on 429 responses. Store the last successfully processed block so the job can resume after an interruption. For high-volume contracts, consider running your own node or using a paid RPC tier with higher limits rather than relying on a public endpoint.