Backfilling ERC‑20 Transfer Events with web3.js: A Practical Guide

Why a single‑shot getPastLogs call almost never works

When you try to fetch the entire Transfer history of a popular ERC‑20 token with a single web3.eth.getPastLogs request, most public RPCs either time out, reject the request, or return only a fragment. Providers impose limits on the number of blocks that can be queried in one call (often 10,000–100,000 blocks) and on the size of the response payload. Additionally, the chain head is still subject to reorgs, so even a “latest” query can return blocks that later disappear.

To build a production‑grade backfill, treat the job as an idempotent cursor that walks the chain in small, confirmed windows. Each window is decoded, persisted, and only then does the cursor advance. This guarantees that a failure halfway through a window does not corrupt the state and that a reorg can be handled cleanly.

Designing the Backfill Job

• Checkpoint – The last fully processed block number. Store it in a durable key‑value store (Redis, DB, or a file).
• Confirmation Buffer – Subtract a small number of blocks (e.g., 12) from latest to avoid reorg‑affected blocks.
• Window Size – Choose a size that stays well below your provider’s limits (e.g., 5,000 blocks).
• Unique Key for Logs – Use transactionHash + logIndex to deduplicate and to recover from reorgs.
• Retry Strategy – If a window fails, halve its size and retry until it succeeds or hits a minimal size.

Implementation Details

Below is a minimal, version‑agnostic example that works with web3.js v1 and v4. Replace the placeholders with your own values. The code is written for Node.js and uses async/await.

// 1️⃣  Import and instantiate
// For v1: const Web3 = require('web3');
// For v4: import Web3 from 'web3';
const web3 = new Web3('https://mainnet.infura.io/v3/YOUR-PROJECT-ID');

// 2️⃣  Contract details
const contractAddress = '0xYourTokenAddress';
const eventSignature = 'Transfer(address,address,uint256)';
const topic0 = web3.eth.abi.encodeEventSignature(eventSignature); // keccak256 of the signature

// 3️⃣  Backfill parameters
const windowSize = 5000;          // blocks per request
const confirmationBuffer = 12;     // blocks to wait for finality
const checkpointKey = 'erc20_transfer_checkpoint'; // key in your persistence layer

// 4️⃣  Helper to load/store checkpoint (pseudo‑code)
async function getCheckpoint() { /* read from DB/Redis */ }
async function setCheckpoint(blockNumber) { /* write to DB/Redis */ }

// 5️⃣  Main loop
async function backfill() {
  let checkpoint = await getCheckpoint(); // e.g., 0 if first run
  const latest = await web3.eth.getBlockNumber();
  const safeHead = latest - confirmationBuffer;

  while (checkpoint < safeHead) {
    const fromBlock = checkpoint + 1;
    const toBlock = Math.min(fromBlock + windowSize - 1, safeHead);

    try {
      const logs = await web3.eth.getPastLogs({
        address: contractAddress,
        topics: [topic0],
        fromBlock,
        toBlock
      });

      // 6️⃣  Process logs
      for (const log of logs) {
        const decoded = web3.eth.abi.decodeLog(
          [{ name: 'from', type: 'address', indexed: true },
           { name: 'to', type: 'address', indexed: true },
           { name: 'value', type: 'uint256', indexed: false }],
          log.data,
          log.topics.slice(1) // skip topic0
        );
        // Persist decoded event + unique key
        const uniqueKey = `${log.transactionHash}-${log.logIndex}`;
        // storeEvent(uniqueKey, decoded, log.blockNumber, log.transactionIndex);
      }

      // 7️⃣  Advance checkpoint
      await setCheckpoint(toBlock);
      checkpoint = toBlock;
    } catch (err) {
      console.warn(`Window ${fromBlock}-${toBlock} failed: ${err.message}`);
      // Reduce window size on failure
      if (windowSize / 2 < 100) throw err; // minimal window size
      windowSize = Math.floor(windowSize / 2);
    }
  }
}

backfill().catch(console.error);

**Key points in the example**

• Logs are filtered by address and topic0 only, which is efficient and reliable.
• All numeric values from the provider are returned as hex strings. web3.eth.abi.decodeLog handles conversion, but if you compare block numbers yourself, convert with web3.utils.hexToNumber or BigInt.
• The checkpoint is advanced only after the entire window is persisted, making the job idempotent.
• On failure the window size is halved. This simple back‑off strategy prevents hammering rate‑limited endpoints.

Reorgs, Archive Endpoints, and Version Nuances

Reorgs invalidate blockNumber alone. By storing transactionHash and logIndex you can detect when a log has been orphaned. If a reorg occurs, your system should:

1. Detect the fork point by comparing stored block hashes with the current chain.
2. Rollback the checkpoint to the fork block minus one.
3. Re‑process from that point. Because the job is idempotent, duplicate logs are harmless.

Many public RPCs (e.g., Infura free tier) are not archive nodes, so eth_getLogs will only return logs from the last ~3‑5 years. For older history, use a dedicated archive provider or run your own full node.

When migrating between web3.js v1 and v4, note:

• Import syntax changes (require vs import).
• v4 uses BigInt for numeric fields by default; v1 uses BN or hex strings.
• Callback style is deprecated in v4; use promises/async.

Trade‑offs & Limitations

• Throughput vs. Reliability – Larger windows speed up backfill but increase the chance of provider rejection. Tune windowSize to your provider’s limits.
• Storage Overhead – Persisting every log’s unique key prevents duplicates but consumes space.
• Reorg Window – A 12‑block confirmation buffer is a common practice, but the optimal size depends on network stability.
• Provider Policy – Some endpoints throttle or block repeated eth_getLogs calls. Implement exponential back‑off and respect Retry-After headers if available.

Actionable Next Steps

1. Run a single small getPastLogs request against your provider to confirm the shape of the response (blockHash, logIndex, topics).
2. Set up the checkpoint store and test the loop with a minimal windowSize (e.g., 100 blocks).
3. Measure the time per window and adjust windowSize and confirmationBuffer accordingly.
4. Deploy the backfill as a background worker (e.g., Bull, Agenda) so it can recover from crashes and run continuously.
5. Implement a reorg detector that watches blockHash changes and triggers a rollback if needed.

By following this pattern you can reliably backfill historic ERC‑20 Transfer logs, handle provider limitations, and maintain idempotent state even in the face of chain reorganizations.