High latency observed when using Mongoose skip/limit for deep page numbers
0 reputation · 05 Dec 2020, 15:27 UTC
Goal: Determine why query response time grows noticeably when requesting higher page numbers while paginating a large collection with Mongoose’s skip() and limit() methods.
Constraints: The collection contains millions of documents, the application must return results in a consistent order, and we cannot change the existing sort field without impacting other services. It is unclear whether the observed latency stems from the server‑side scan caused by large skip values or from client‑side memory usage when streaming results.
Specific questions:
- Does the
docsExaminedmetric increase roughly asskip + limitfor these queries, confirming a linear scan cost? - Would replacing
skip/limitwith a range query on an indexed monotonic field (e.g.,_id) eliminate the scan overhead while preserving order? - When using
Query.cursor()to stream results, what batch size should be set to keep memory usage bounded without sacrificing throughput?