Paginating Twitter API v2 Recent Search with next_token While Staying Inside Rate Limits

The problem: pulling thousands of recent tweets without hitting the 450-request ceiling

Developers who need a large slice of recent tweets quickly discover that a single call returns at most 100 results. The API therefore forces you to page through the data using an opaque next_token. If you blindly loop, you'll exhaust the 450-request-per-15-minute window (elevated access) long before you've collected the dataset you need.

How next_token pagination works

The /2/tweets/search/recent endpoint returns a JSON payload that always contains a meta object. The fields you care about are:

• result_count – number of tweets in this page (≤ 100).
• next_token – a string you must append as a query parameter to fetch the following page. Treat it as opaque; do not parse or store it for later reuse.
• newest_id / oldest_id – optional bounds useful for debugging but not required for pagination.

When next_token is absent or null, you have reached the end of the result set for the supplied query.

Rate-limit budgeting

Elevated access grants 450 requests per rolling 15-minute window. With max_results=100 you can theoretically retrieve 45,000 tweets per window. The response headers expose the current budget:

• x-rate-limit-limit – 450
• x-rate-limit-remaining – requests left in the current window
• x-rate-limit-reset – epoch seconds when the window resets

A safe client reads x-rate-limit-remaining after each response and pauses (or backs off) when the count drops below a comfortable threshold (e.g., 20).

Worked example: fetching 5,000 tweets from @twitterdev

Assume you have a bearer token with elevated scope. Run the following from a terminal that has curl installed (Linux/macOS/WSL). Replace <BEARER> with your token.

# 1️⃣ First request – ask for 100 tweets per page
curl -G "https://api.twitter.com/2/tweets/search/recent" \
  -H "Authorization: Bearer <BEARER>" \
  --data-urlencode "query=from:twitterdev" \
  --data-urlencode "max_results=100" \
  -v

Inspect the response body for meta.next_token and the headers for x-rate-limit-remaining. Example snippet of the JSON you'll see:

{
  "data": [ { "id": "176...", "text": "..." }, … ],
  "meta": {
    "result_count": 100,
    "next_token": "b26v89c19zqg8…",
    "newest_id": "176…",
    "oldest_id": "175…"
  }
}

2️⃣ Pull the next page by feeding the token back:

TOKEN="b26v89c19zqg8…"   # copy from previous output
curl -G "https://api.twitter.com/2/tweets/search/recent" \
  -H "Authorization: Bearer <BEARER>" \
  --data-urlencode "query=from:twitterdev" \
  --data-urlencode "max_results=100" \
  --data-urlencode "next_token=$TOKEN" \
  -v

Repeat until next_token is missing or you have accumulated the desired tweet count. A tiny Bash loop that respects the rate-limit header looks like:

#!/usr/bin/env bash
BEARER="<BEARER>"
QUERY="from:twitterdev"
MAX=100
TOKEN=""
TOTAL=0
while :; do
  ARGS=(--data-urlencode "query=$QUERY" --data-urlencode "max_results=$MAX")
  [[ -n $TOKEN ]] && ARGS+=(--data-urlencode "next_token=$TOKEN")
  RESP=$(curl -s -G "https://api.twitter.com/2/tweets/search/recent" \
    -H "Authorization: Bearer $BEARER" \
    "${ARGS[@]}" -w "\n%{http_code} %{header_x-rate-limit-remaining} %{header_x-rate-limit-reset}")
  BODY=$(echo "$RESP" | head -n -1)
  META=$(echo "$BODY" | jq -r '.meta')
  COUNT=$(echo "$META" | jq -r '.result_count')
  TOKEN=$(echo "$META" | jq -r '.next_token // empty')
  TOTAL=$((TOTAL + COUNT))
  echo "Fetched $COUNT tweets (total $TOTAL)"
  REM=$(echo "$RESP" | tail -n1 | awk '{print $2}')
  if (( REM < 20 )); then
    RESET=$(echo "$RESP" | tail -n1 | awk '{print $3}')
    SLEEP=$((RESET - $(date +%s) + 5))
    echo "Rate limit low, sleeping $SLEEP seconds"
    sleep $SLEEP
  fi
  [[ -z $TOKEN ]] && break
  [[ $TOTAL -ge 5000 ]] && break
done

What to verify

• Each response returns HTTP 200 and a new next_token (or none at the end).
• x-rate-limit-remaining decrements by one per request.
• When the token expires (≈ a few minutes of inactivity), the API replies 400 with {"detail":"Invalid next_token provided"}. The script above would need to restart from the first request.

Trade-offs and limitations

Aspect	Impact
Token lifetime	Opaque tokens expire after a short idle period; you cannot persist them across runs.
Maximum results per call	Hard-capped at 100; you cannot request larger pages to reduce request count.
Rate-limit window	450 requests per 15 min; burst-heavy workloads must implement back-off or spread across multiple bearer tokens (if you have multiple elevated apps).
Result set volatility	Because the recent index is a moving window (last 7 days), the same query can return different tweets between pages, especially near the oldest edge.

Actionable checklist for production code

1. Obtain an elevated-access bearer token and store it securely (env var, secret manager).
2. Set max_results=100 to maximise throughput per request.
3. After each response, read x-rate-limit-remaining; if < 20, sleep until x-rate-limit-reset + a small buffer.
4. Treat next_token as transient – discard on error 400 and restart pagination.
5. Persist only the tweet IDs or payload you need; do not cache the token.
6. Log the newest_id/oldest_id for auditability and to detect gaps.

Following this pattern lets you reliably harvest tens of thousands of recent tweets while staying inside Twitter's documented limits.