Cursor expiration during paginated metric queries with extended time ranges
0 reputation · 06 Mar 2020, 04:13 UTC
Context
Datadog's Metrics API uses cursor-based pagination for large dataset retrieval, returning a cursor token in each response to fetch the next page. The documentation indicates cursors expire if the interval between paginated requests grows too large, but the exact expiration window is not publicly specified.
Problem
When querying metrics across extended time ranges (e.g., 30+ days) with high-cardinality tag filters, the number of pages can exceed several hundred. Automated pagination loops must respect rate limits (monitoring X-RateLimit-Remaining headers) while also completing before the cursor becomes invalid. The interaction between rate-limit pacing and cursor lifetime creates an unbounded failure mode: slowing down to avoid 429 responses increases the risk of cursor expiration, while speeding up risks rate-limit rejection.
Unresolved behavior
It is unclear whether the cursor TTL is fixed, scales with the original query time range, or depends on backend load. No documented error code distinguishes cursor expiration from other pagination failures, making automated retry logic difficult to design.
What is the documented or observable cursor TTL for the Metrics API pagination endpoint? Does the TTL vary based on the from/to time range or result set size? Is there a distinct error response or header that signals cursor expiration versus a generic pagination failure?