Choosing a Pagination Strategy for X API v2 Timelines and Search

If you need every tweet matching a query or every post on a timeline, the X API v2 gives you three workable patterns — cursor pagination, time-windowed iteration, and the filtered stream — and they solve different problems. Picking the wrong one usually means either silently missing data or building parallelism you never needed. This guide compares the options and shows a concrete, resumable implementation you can validate.

One caveat before the details: access tiers, endpoint availability, and pricing for the X API have changed repeatedly. Verify your app's tier and the current documented limits in the developer portal before building on anything below.

The decision and its constraints

The core constraint is that X API v2 list endpoints (user timelines, recent search, full-archive search) are cursor-based. There is no offset parameter. Each response includes a meta.next_token value, and you fetch the next page by passing it back as pagination_token. You cannot jump to "page 40" or synthesize your own cursors.

Other constraints that shape the decision:

• max_results is bounded per endpoint (commonly 5–100 for timelines and search). The API may return fewer results than requested, so a short page does not mean you've reached the end. Only the absence of meta.next_token terminates the loop.
• Recent search covers roughly the last 7 days. Older data requires full-archive search, which is gated by access tier.
• Rate limits are per-endpoint and per app or user context. A naive paginator that retries immediately on HTTP 429 will stall or get suspended.
• Tweet objects return minimal fields by default. Author, media, and metrics require explicit tweet.fields and expansions parameters — forgetting this is a common cause of "missing data" that has nothing to do with pagination.

Comparing the three options

The practical rule: use plain cursor pagination for anything that finishes in one sitting, add time windows when the job is big enough that a serial crawl is too slow or too fragile, and use the stream only when "history" is not part of the requirement. Streaming and paging are complements, not alternatives — many pipelines backfill with windows and then keep current with a stream.

Trade-offs that bite in practice

Serial vs. parallel. Cursor paging is one request at a time by construction. Time-windowed backfill lets you shard windows across workers and retry a single failed window without redoing the whole range. The cost is bookkeeping: choose window boundaries carefully (align to whole minutes or hours), make windows slightly non-overlapping, and dedupe on tweet ID anyway, because a tweet exactly on a boundary can appear in two windows.

Resumability. Either paging approach should persist state. For cursor paging, store the last next_token per job. For windows, store which (start, end) pairs are complete. Without this, a rate-limit pause or process restart means starting over.

Rate limits. Centralize request scheduling in one place rather than scattering sleep calls. Read the rate-limit response headers (x-rate-limit-remaining, x-rate-limit-reset) and, on a 429, wait until the reset time instead of retrying immediately. Immediate retries compound the problem.

A concrete implementation: resumable recent-search pull

The pattern below (Python, run anywhere with network access and a bearer token; no special permissions beyond your app's access tier) pages through recent search with a persisted cursor. Placeholders: YOUR_BEARER_TOKEN and YOUR_QUERY.

import json, os, time, requests

TOKEN = "YOUR_BEARER_TOKEN"
URL = "https://api.twitter.com/2/tweets/search/recent"
STATE_FILE = "cursor_state.json"

def load_cursor():
    if os.path.exists(STATE_FILE):
        return json.load(open(STATE_FILE)).get("next_token")
    return None

def save_cursor(tok):
    json.dump({"next_token": tok}, open(STATE_FILE, "w"))

headers = {"Authorization": f"Bearer {TOKEN}"}
params = {
    "query": "YOUR_QUERY",
    "max_results": 100,
    "tweet.fields": "created_at,author_id",
}
token = load_cursor()
seen_ids = set()

while True:
    if token:
        params["pagination_token"] = token
    r = requests.get(URL, headers=headers, params=params, timeout=30)
    if r.status_code == 429:
        reset = int(r.headers.get("x-rate-limit-reset", time.time() + 60))
        time.sleep(max(reset - time.time(), 1) + 1)
        continue
    r.raise_for_status()
    body = r.json()
    for t in body.get("data", []):
        if t["id"] in seen_ids:
            print("duplicate across pages:", t["id"])
        seen_ids.add(t["id"])
    token = body.get("meta", {}).get("next_token")
    save_cursor(token)
    if not token:
        break

print("total unique tweets:", len(seen_ids))

Key behaviors: the loop ends only when next_token is absent, the cursor is written after every page so a killed process resumes where it stopped, and 429s wait for the documented reset rather than hammering the endpoint. Risk to note: the state file is a single point of truth — if you parallelize later, move state into a shared store and switch to windowed shards.

Validating completeness

Don't trust a clean exit as proof you got everything. Two cheap checks:

1. Duplicate audit. Collect all tweet IDs in a set as above. Any duplicate across pages signals a bug (usually reusing a stale pagination_token or overlapping windows).
2. Cross-check with a different split. Re-run the same query using a different time-window decomposition (e.g., one big window vs. daily windows) and compare the resulting ID sets. They should match exactly. A mismatch almost always means a window boundary issue or an unhandled short page.

Also log x-rate-limit-remaining during a real run and confirm your observed throughput matches the documented limit for your tier — if you're being throttled harder than expected, your backoff logic, not the API, is the first thing to inspect.

Limitations

• Recent search's ~7-day horizon means this pattern cannot backfill older data; that requires full-archive search at the appropriate access tier.
• Results can shift under you: tweets deleted or accounts suspended mid-crawl disappear from later pages, so a second pass may legitimately return slightly fewer IDs. Treat small deltas as expected, large ones as bugs.
• Endpoint limits and tier capabilities change; verify max_results bounds and rate limits in the developer portal for your app before relying on the numbers here.