Handling Pagination and Rate Limits in the X (Twitter) API v2

The Challenge of the Infinite Scroll

When building a client to fetch a user's timeline via the /2/users/:id/tweets endpoint, the primary technical hurdle isn't the first request—it's the subsequent ones. Unlike traditional REST APIs that use numeric page offsets (e.g., page=2), the X API uses cursor-based pagination. If your application assumes a static list of tweets or tries to calculate offsets manually, you will encounter duplicate data or missing entries as new tweets are posted in real-time.

The core takeaway: To reliably traverse a timeline, you must treat the next_token as a temporary pointer to a specific position in the data stream, not a permanent page marker.

Managing Cursor-Based Pagination

The API returns a meta object alongside the data array. If more tweets are available, this object contains a next_token. To retrieve the next set of results, this token must be passed back in the subsequent request as a query parameter.

Crucially, these tokens are not snapshots. Because the timeline is dynamic, a tweet might be deleted or a user's privacy settings might change between your first and second request. This means the next_token identifies a relative position; it does not guarantee that the total count of tweets remains constant during your pagination loop.

Configuration and Request Parameters

When requesting tweets, use these key parameters to control the payload:

• max_results: Defines the number of tweets per page. While you can request a specific number, the API enforces a hard upper limit per request. If you provide an invalid value, the API typically falls back to a default rather than returning an error.
• expansions: By default, you only get basic tweet data. To get public_metrics (likes, retweets), you must explicitly request them via expansions. A missing metrics object in the response does not mean the count is zero; it means the data wasn't requested.

Implementation Example: The Pagination Loop

The following logic demonstrates how to handle the token exchange. This example assumes you are using a language with a standard HTTP client and possess a valid Bearer Token with the necessary access tier permissions.

# Logical flow for fetching a full timeline
# Required Permissions: User Authentication or App-only Bearer Token

user_id = "123456789"
next_token = None
all_tweets = []

while True:
    # Construct URL with optional next_token
    url = f"https://api.twitter.com/2/users/{user_id}/tweets?max_results=100"
    if next_token:
        url += f"&pagination_token={next_token}"

    response = http_client.get(url, headers={"Authorization": "Bearer YOUR_TOKEN"})

    if response.status_code == 429:
        # Handle Rate Limit: Extract 'x-rate-limit-reset' header to determine wait time
        wait_time = response.headers.get("x-rate-limit-reset")
        sleep(wait_time)
        continue

    data = response.json()
    all_tweets.extend(data.get("data", []))

    # Check if more pages exist
    next_token = data.get("meta", {}).get("next_token")
    if not next_token:
        break

Handling Rate Limits and 429 Errors

The X API enforces strict rate limits based on your access tier (Free, Basic, Pro, or Enterprise). When you exceed these limits, the server returns an HTTP 429 Too Many Requests response.

To handle this without crashing your integration, your client must implement a "back-off" strategy. The response headers typically include x-rate-limit-reset, which provides a Unix timestamp indicating when the limit resets. Your application should parse this value and pause execution until that time has passed.

Trade-offs and Limitations

While cursor-based pagination is efficient for the server, it introduces specific constraints for the developer:

• No Random Access: You cannot jump directly to "page 10." You must fetch pages 1 through 9 sequentially to obtain the required token.
• Token Expiration: Pagination tokens are not permanent. If there is a significant delay between requests, the token may expire or become invalid, requiring you to restart the fetch process from the first page.
• Caching Risks: Caching full API payloads can violate developer policies regarding data redistribution. It is generally safer to cache the id of the last processed tweet to detect duplicates rather than storing the entire response body long-term.

Verification Steps

To verify your implementation is working correctly, perform these checks in your developer console or via cURL:

1. Request the first page of a public user's timeline and verify the meta.next_token is present.
2. Perform a second request using that token and confirm the created_at timestamps of the new tweets are older than those from the first page.
3. Verify that you are not receiving duplicate tweet IDs across the page boundary.