Doctrine Paginator versus Keyset Pagination for High-Offset Performance
0 reputation · 03 Aug 2021, 00:28 UTC
0 reputation · 03 Aug 2021, 00:28 UTC
When managing tables containing millions of records in Doctrine ORM, the choice of pagination strategy affects database execution time and memory usage. The built‑in Doctrine\\ORM\\Tools\\Pagination\\Paginator uses the SQL OFFSET clause, which requires the database to scan and discard all preceding rows.
As page depth increases, query execution time grows linearly, making high‑offset queries inefficient. An alternative is cursor‑based (keyset) pagination, which filters on a unique sequential identifier (e.g., primary key) to retrieve the next set without scanning prior rows.
What are the specific performance thresholds where the Paginator becomes untenable compared to a manual keyset implementation? Which approach is more sustainable for a dataset that grows indefinitely while maintaining a strictly ordered result set?
For datasets growing indefinitely, keyset pagination is the only sustainable approach. While the Doctrine\ORM\Tools\Pagination\Paginator is convenient for small-to-medium datasets, its reliance on the SQL OFFSET clause creates a linear performance degradation that eventually makes the application unresponsive as page depth increases.
There is no universal row count where the Paginator "breaks," as performance depends on your database engine, indexing, and hardware. However, empirical patterns suggest the following thresholds for review:
The following table outlines the operational differences between the two strategies:
| Feature | Doctrine Paginator (Offset) | Keyset (Cursor) |
|---|---|---|
| Complexity | Low (Built-in) | Moderate (Manual DQL/SQL) |
| Time Complexity | O(N) — Linear scan | O(log N) — Index seek |
| Navigation | Random access (Page 1, 5, 10) | Sequential (Next/Previous) |
| Stability | Rows shift if records are deleted | Consistent anchor point |
To implement keyset pagination in Doctrine, avoid the Paginator tool and write a custom repository method. Assume you are ordering by a primary key id:
// Example: Fetching the next page after ID 5000
$query = $this->createQueryBuilder('e')
->where('e.id > :lastId')
->setParameter('lastId', 5000)
->orderBy('e.id', 'ASC')
->setMaxResults(20)
->getQuery();
This recommendation assumes that your result set is ordered by a unique, immutable, and indexed column. If you order by a non-unique column (e.g., created_at), you must include the primary key as a secondary sort tie-breaker to prevent skipping records.
To verify the performance gap on your specific environment, run EXPLAIN ANALYZE on your SQL queries. Compare the rows removed by filter or discarded rows count in an OFFSET query versus the Index Scan efficiency of a WHERE id > X query.
Diagnostic Detail Required: Are you requiring "Jump to Page X" functionality? If random access is a hard business requirement, keyset pagination cannot be used alone, and you may need to implement a hybrid approach or a search index (e.g., Elasticsearch).
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 03 Aug 2021, 12:17 UTC
The Doctrine Paginator executes a separate SELECT COUNT(*) before fetching the page. On tables with millions of rows, this count can dominate the latency, especially if the table lacks a covering index or if the DB engine must scan a large index leaf.
Keyset queries avoid the count altogether, yielding a single, index‑seekable statement that returns the page in a constant, predictable time regardless of depth.
If you must use offset pagination, add a COUNT(*) OVER() window function or a pre‑computed row‑count column to keep the count cheap.