X API v2 recent-search pagination: handling pages that exactly match max_results

Short answer

No. A page that returns exactly max_results items does not guarantee the presence of meta.next_token. If the matching dataset happens to end exactly on a page boundary, the final page can be full and still have no next_token. The only authoritative signal that more pages exist is the presence of meta.next_token in the response — never the item count.

How to decide whether to keep paginating

The common logic error is writing a loop like while len(items) == max_results. That condition is wrong in both directions:

• A full page can be the last page (dataset ends exactly at the boundary) — you'd fire one extra request that returns zero or few results with no token.
• Relying on count also can't distinguish "exactly full and done" from "exactly full with more coming" — only the token can.

The correct loop is token-driven:

token = None
while True:
    resp = search_recent(query, max_results=100, next_token=token)
    yield resp["data"]
    token = resp.get("meta", {}).get("next_token")
    if not token:
        break

When next_token is absent, pagination is complete — regardless of how many items the last page contained.

Token validity and expiration

X does not publish a guaranteed validity duration for pagination_token values. Treat them as short-lived, opaque cursors tied to your exact query state, not permanent pointers. Practical implications:

• Use a token promptly after receiving it; don't persist tokens across long-running jobs or process restarts and expect them to work later.
• Do not modify any query parameters (query string, start/end time, max_results semantics) between paginated requests — tokens are bound to the original request parameters, and changing them invalidates the token.
• An expired or invalid token typically surfaces as a 400-class Bad Request error, which can look like a malformed-request or data problem. If a paginated request fails with 400 after earlier pages succeeded, suspect token expiry or parameter drift before suspecting your query syntax.

Verifying the behavior yourself

1. Run a query with a small max_results (e.g., 10) against a high-volume term. Confirm meta.next_token appears while more results exist.
2. Find a query whose total match count is an exact multiple of your max_results (or approximate it by paging to the end) and observe that the last full page arrives without a next_token.
3. Capture a token, wait an extended period (hours to a day), and replay it to observe the actual expiration error in your environment — since no official TTL is documented, this is the only reliable way to know the effective lifetime.

Caveats

This reflects well-established v2 pagination behavior as generally documented and observed, but X's API terms and endpoint behavior change periodically. Confirm against the current developer documentation for your access tier, and treat the token-lifetime question as environment-specific until you've measured it yourself.