Cursor vs. Offset Pagination in Harvester GraphQL: Which Keeps Data Consistent When Records Change?

Goal: Retrieve a consistent snapshot from a large GraphQL dataset

The Harvester GraphQL connector offers cursor‑based pagination via the first and after arguments, and it also provides date‑range filters such as createdAfter and createdBefore to bound queries. When the underlying dataset is subject to insertions, updates, or deletions between paginated requests, the chosen pagination strategy can affect whether duplicates or gaps appear in the harvested data.

Constraints and Uncertainty

Two documented approaches exist: cursor‑based pagination, which relies on a stable ordering and records the last cursor value, and offset pagination, which uses a numeric page index. The Harvester documentation does not specify a fallback from cursor to offset, nor does it quantify the performance or consistency impact of each method when the dataset changes between fetches. The core uncertainty is how Harvester reconciles such changes—does it assume a stable order, or does it require a full re‑fetch to maintain data integrity?

Key Questions

1. Does Harvester guarantee no duplicates or gaps when new records are inserted between paginated calls using cursor pagination?
2. What is the performance impact if the underlying GraphQL API does not provide a stable ordering for cursor pagination?
3. Is there a recommended strategy for handling dataset mutations between pages—re‑fetching the entire set or switching to offset pagination?