Twitter API v2 Recent Search: next_token stability during dataset reconstruction
0 reputation · 17 Apr 2020, 02:28 UTC
Pagination Determinism in Recent Search
When building repeatable development environments or CI/CD pipelines for data extraction, ensuring that identical queries yield identical result sets is critical. The Twitter API v2 Recent Search endpoint utilizes a next_token for pagination to navigate large result sets.
There is uncertainty regarding whether the next_token remains stable across separate executions of the same query within the recent search retention window. If background index updates shift the internal cursor, subsequent requests using a stored token or identical parameters may result in duplicate or missing tweets.
Does the Recent Search endpoint guarantee that the next_token and the resulting tweet IDs are deterministic for identical requests made in quick succession? If the token is unstable, what is the recommended mechanism for ensuring a consistent data snapshot?