Paginating X API v2 Recent Search with next_token: A Reliable Retrieval Guide

Desired Outcome

Retrieve every tweet matching a query from the last ~7 days by walking the next_token cursor until the response meta object no longer contains a token. The guide covers prerequisites, the exact request sequence, validation checks on each page, and recovery when tokens expire or rate limits hit.

Prerequisites

• X Developer account with a Project and App created on the X Developer Platform.
• Authentication credential: either an app-only Bearer Token (OAuth 2.0 Client Credentials) or a user-context OAuth 2.0 access token with tweet.read scope.
• Query string built from supported recent-search operators (e.g., from:twitterdev -is:retweet lang:en).
• Awareness of the lookback window: recent search covers approximately the last seven days. The boundary is enforced server-side and is not exact; do not design around a precise cutoff.
• Access tier confirmation: rate limits, max_results bounds, and available expansions/fields vary by tier (Free, Basic, Pro, Enterprise). Verify current quotas in the official documentation before coding.

Procedure: Page Walk with next_token

The recent search endpoint is GET /2/tweets/search/recent. Every paginated request must be identical except for the next_token parameter. Changing query, max_results, expansions, tweet.fields, user.fields, or any other parameter between pages will invalidate the cursor and return a 400 error.

First Request

curl -G "https://api.x.com/2/tweets/search/recent" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  --data-urlencode "query=from:twitterdev -is:retweet lang:en" \
  --data-urlencode "max_results=100" \
  --data-urlencode "tweet.fields=created_at,author_id,public_metrics" \
  --data-urlencode "expansions=author_id"

Where to run: any shell with curl and the BEARER_TOKEN environment variable set. The token must have tweet.read permission. No elevated OS privileges are required.

Placeholders: replace $BEARER_TOKEN with your actual token. Adjust query, max_results (10–100 per request depending on tier), and field parameters to your needs.

Response Structure

The JSON body contains two top-level objects:

• data: array of tweet objects (empty array if no matches).
• meta: pagination metadata including result_count (number of tweets in this page), next_token (opaque string for the next page), and optionally newest_id, oldest_id.

Example minimal response:

{
  "data": [
    {"id": "1798765432109876543", "text": "...", "created_at": "2024-06-10T12:34:56.000Z", "author_id": "2244994945"}
  ],
  "meta": {
    "result_count": 1,
    "next_token": "b26v89c19zqg8y3fpk2h3g8o9v3z...",
    "newest_id": "1798765432109876543",
    "oldest_id": "1798765432109876543"
  }
}

Subsequent Pages

If meta.next_token is present, issue the exact same request with the token appended:

curl -G "https://api.x.com/2/tweets/search/recent" \
  -H "Authorization: Bearer $BEARER_TOKEN" \
  --data-urlencode "query=from:twitterdev -is:retweet lang:en" \
  --data-urlencode "max_results=100" \
  --data-urlencode "tweet.fields=created_at,author_id,public_metrics" \
  --data-urlencode "expansions=author_id" \
  --data-urlencode "next_token=b26v89c19zqg8y3fpk2h3g8o9v3z..."

Repeat until a response meta lacks next_token. That page is the last one.

Expected Checks on Every Page

1. Count integrity: meta.result_count must equal data.length. A mismatch indicates a malformed response or partial failure.
2. Deduplication: Tweet IDs (data[i].id) must be unique across all accumulated pages. Defensively store seen IDs in a set and log any collision—this should never happen with a valid cursor but protects against logic errors.
3. Termination condition: The final page has no next_token in meta. Do not issue another request.
4. Accumulated count sanity: After the walk, compare the total collected tweets against a rough expectation for the query and time window. Large deviations suggest a query issue or an incomplete lookback.

Recovery Options

Expired or Invalid next_token (HTTP 400)

The token is opaque and short-lived (typically minutes). If you receive a 400 with an error like "Invalid or expired token", restart pagination from the first page using the original query and parameters. Do not attempt to reuse or guess a token.

Rate Limit (HTTP 429)

Inspect the response headers:

• x-rate-limit-limit: requests allowed in the current window.
• x-rate-limit-remaining: requests left.
• x-rate-limit-reset: Unix epoch seconds when the window resets.

Pause until x-rate-limit-reset (plus a small buffer), then resume from the page you were on—the same next_token remains valid across a rate-limit pause as long as the token hasn't expired.

Authentication Failure (HTTP 401/403)

• 401: Bearer token is missing, malformed, or revoked. Refresh the token (re-run OAuth flow or regenerate in the developer portal).
• 403: Token lacks required scope or the App/Project does not have recent-search access for your tier. Check the App's permissions and your tier's entitlements.

Engineering Decision: Don't Persist next_token for Continuous Collection

next_token is tied to one exact request shape and expires quickly. For ongoing ingestion, do not checkpoint the token. Instead, checkpoint:

• The query string.
• The time window (e.g., start_time/end_time if your tier supports them, or a rolling 7-day window).
• The newest_id (or since_id) of the last successfully processed tweet.

Run scheduled recent-search jobs per window (e.g., every 15 minutes) using since_id to pick up where you left off. This avoids token-expiry restarts and works reliably across deployments.

Complementary Architecture: Backfill + Stream

For low-latency needs, use the Filtered Stream endpoint (GET /2/tweets/search/stream) with stored rules. Reserve recent-search pagination for:

• Historical backfill when starting a new rule.
• Scheduled windowed analyses (e.g., daily rollups).
• One-off investigations.

The two combine into a backfill-plus-stream architecture: stream for real-time, recent-search for catch-up and batch.

Limitations & Verification

• Lookback is approximate: The 7-day window is enforced server-side and may vary by a few hours. Test by querying a known tweet ID older than 7 days; recent search should not return it.
• Full-archive search requires Elevated/Enterprise tier and uses a different endpoint (/2/tweets/search/all).
• Expansions and fields (e.g., referenced_tweets, geo) may be restricted by tier. Confirm availability in the docs for your access level.
• Rate limits are tier-specific. Run a two-page smoke test and log x-rate-limit-* headers to observe your actual quota.

Smoke Test Checklist

1. Fetch page one with your query and max_results=10.
2. Capture meta.next_token.
3. Fetch page two with that token.
4. Assert: meta.result_count === data.length on both pages.
5. Assert: No duplicate id values across the two pages.
6. Assert: Page two either has a new next_token or none (end of results).
7. Log x-rate-limit-remaining and x-rate-limit-reset from both responses.

If all assertions pass, the pagination loop is wired correctly for your tier and query.