Consuming Notion's Cursor-Based Pagination Safely

The problem: pagination that looks simple but bites you in production

You start querying a Notion database with the API. The first 100 rows come back fast. You ship the integration. Weeks later a user adds row 101 and your sync misses it—because you assumed a single request was enough. Notion's API uses cursor-based pagination on every list endpoint: database queries, block children, search. The mechanism is stable, but the surrounding constraints (rate limits, recursive block trees, eventual consistency) turn a naive loop into a reliability hazard.

How Notion's pagination actually works

Every list endpoint returns a JSON object with three keys:

• results: array of objects for the current page
• has_more: boolean, true when another page exists
• next_cursor: opaque string to pass as start_cursor on the next request, or null when done

You control page size with the page_size parameter (max 100). The cursor is opaque—do not parse, construct, or cache it beyond the immediate pagination sequence. Treat it as a token the server understands and your client merely forwards.

A robust consumption pattern

The safe pattern is a generator (or async iterator) that yields each item while handling three cross-cutting concerns:

1. Loop until has_more is false – not until the results array is empty; a page can legitimately return zero items while has_more stays true.
2. Throttle and retry on HTTP 429 – Notion enforces rate limits (commonly a few requests per second per integration). Exponential backoff with jitter prevents hammering the endpoint.
3. Deduplicate by object ID – because content can shift between requests (rows added, deleted, reordered), the same ID may appear on two consecutive pages. A Set of seen IDs is cheap insurance.

Worked example: Python generator with backoff

The following snippet runs in a script or background worker. It assumes a notion_client with a post method that raises on non-2xx responses. Adjust MAX_PAGE_SIZE and BASE_DELAY to your integration's quota.

import time
import random
from typing import Iterator, Any, Dict

def paginate_database(
    notion_client,
    database_id: str,
    page_size: int = 50,
    filter_: Dict | None = None,
    sorts: list[Dict] | None = None,
) -> Iterator[Dict[str, Any]]:
    """Yield every row in a database, handling pagination, rate limits, and deduplication."""
    url = f"https://api.notion.com/v1/databases/{database_id}/query"
    payload: Dict[str, Any] = {"page_size": min(page_size, 100)}
    if filter_:
        payload["filter"] = filter_
    if sorts:
        payload["sorts"] = sorts

    seen_ids = set()
    has_more = True
    next_cursor = None
    attempt = 0
    BASE_DELAY = 1.0  # seconds

    while has_more:
        if next_cursor:
            payload["start_cursor"] = next_cursor

        try:
            resp = notion_client.post(url, json=payload)
            data = resp.json()
            attempt = 0  # reset on success
        except Exception as e:
            # Assume the client raises a custom RateLimitedError with retry_after
            if hasattr(e, 'retry_after'):
                wait = e.retry_after + random.uniform(0, 0.5)
                time.sleep(wait)
                attempt += 1
                if attempt > 5:
                    raise
                continue
            raise

        for item in data.get("results", []):
            item_id = item.get("id")
            if item_id and item_id not in seen_ids:
                seen_ids.add(item_id)
                yield item

        has_more = data.get("has_more", False)
        next_cursor = data.get("next_cursor")

        # polite delay between pages even when not rate-limited
        time.sleep(BASE_DELAY + random.uniform(0, 0.2))

Key points in the code:

• page_size is capped at 100 locally; the API will reject larger values.
• The seen_ids set prevents duplicate yields if a row appears on two pages.
• Rate-limit handling assumes the HTTP client surfaces a retry_after value (Notion returns a Retry-After header on 429). If your client doesn't, parse the header yourself.
• A small randomized delay between successful pages keeps you well under typical limits.

Recursive block children: the hidden pagination tree

A page's content is a tree of blocks. To render or export a page you must:

1. Call GET /v1/blocks/{block_id}/children with pagination.
2. For each block where has_children is true, recurse into its children endpoint.

This means a single page export can trigger dozens of paginated requests. Apply the same generator pattern at each level, and consider a breadth-first queue to avoid deep recursion limits.

Trade-offs and limitations

• Search is not strongly consistent. Recently created pages may not appear in /v1/search results immediately. If your integration depends on search, add a short delay or fall back to database queries where you control the filter.
• Rate-limit numbers are not contractual. The "few requests per second" figure changes per integration tier and Notion's internal policy. Always read the current Retry-After header rather than hardcoding a delay.
• Cursors expire. A cursor is only valid for the immediate next request. Storing it across process restarts or sharing it between workers will fail.

Verify before you ship

1. Create a test database with 150+ rows. Run the generator with page_size=25 and assert the total yielded count equals the row count.
2. Log every next_cursor and verify no duplicate or missing IDs across the concatenated pages.
3. In a sandbox integration, deliberately exceed the rate limit (e.g., tight loop with zero delay) and confirm your backoff logic recovers cleanly without data loss.

Once those checks pass, you have a pagination layer that survives workspace growth, concurrent edits, and Notion's operational limits.