NSFetchRequest fetchOffset pagination: latency growth as a failure condition for large datasets

Direct answer

There is no fixed dataset size at which fetchOffset becomes unsuitable; the failure condition is a measured latency cliff on your specific stack. The object ID batch pattern preserves predicate and sort semantics only if you re-sort materialized objects in memory, and it adds store round trips. Verification requires a controlled benchmark on a copy of the real store, not a universal threshold.

Confirmed behavior vs. likely explanation

Confirmed: On the SQLite store (the default on Apple platforms), fetchOffset maps to SQL OFFSET semantics. The store evaluates the full NSPredicate and sort ordering, then discards skipped rows, so per-page execution time grows roughly linearly with offset depth.

Likely explanation: Interactive paging becomes janky when the per-page fetch time approaches or exceeds the frame budget. Because the cost scales with offset, early pages are cheap and deep pages are expensive. The exact breaking point depends on predicate complexity, sort-key indexes, row width, device speed, and whether returned objects are faults.

Unconfirmed heuristics: Community reports of trouble at tens of thousands of rows are hypotheses to measure, not confirmed constants. Treat any specific number as a prompt for a benchmark, not a rule.

Object ID batching: semantics and trade-offs

The ID pattern fetches NSManagedObjectID values once using the same predicate and sort descriptors (resultType = .managedObjectIDResultType), then pages by fetching objects with a self IN %@ predicate over ID slices. This avoids repeated offset re-scans because subsequent pages resolve rows by primary key.

• Sort caveat: self IN %@ does not guarantee the original sort order. You must re-order the materialized objects in memory to match the ID array to preserve the same semantics.
• Round trips: The pattern adds one full ID fetch up front plus one object fetch per batch. For shallow offsets or small datasets, this can be slower than a single offset fetch.
• Stability: An ID snapshot from a single fetch is stable under concurrent writes, whereas offset pagination can skip or repeat rows if the store changes between pages.
• Memory: Holding the full ID array costs memory proportional to dataset size; IDs are small but not free on very large stores.

Verification method

1. Run with the launch argument -com.apple.CoreData.SQLDebug 1 to inspect generated SQL and confirm LIMIT/OFFSET appears and its value grows with fetchOffset.
2. Benchmark both approaches on a copy of the real store. Measure per-page fetch duration at increasing offsets (signposts or XCTest measure blocks) versus ID-batch fetches with identical predicate and sort.
3. Use Instruments' Core Data and Time Profiler templates to compare fetch counts and wall-clock time per page as offset depth increases.

One diagnostic that changes the recommendation

What is the maximum offset depth your paging UI reaches before a fetch is triggered? If the UI only ever loads a few pages with shallow offsets, the simpler fetchOffset approach may remain acceptable. If the UI scrolls deep into a large dataset, the ID-batch pattern is likely the better trade-off despite the extra round trips.