$lookup Sub‑Pipeline Cursor Timeout: noCursorTimeout Uncertainty
0 reputation · 16 Dec 2022, 07:26 UTC
Goal
Determine whether the noCursorTimeout option in a $lookup sub‑pipeline reliably prevents cursor expiration during large aggregations.
Context
The aggregation framework allows a $lookup stage to include a sub‑pipeline that runs as a separate query. When the local collection is large, the sub‑pipeline may open a cursor on the foreign collection. The default cursor timeout is 30 seconds, but it is not documented whether this timeout is applied to the sub‑pipeline cursor.
Unresolved Behavior
If the sub‑pipeline cursor times out before the aggregation completes, the pipeline fails with a timeout error, producing incomplete results. This unpredictable behavior can mask performance bottlenecks and lead to incorrect data processing.
Specific Questions
- Does the
noCursorTimeoutflag in the$lookupsub‑pipeline disable the cursor timeout for that cursor? - Under what conditions (e.g., data size, sharding, network latency) does the sub‑pipeline cursor expire during aggregation?
- Is there a server‑level configuration that can globally disable cursor timeouts for aggregation sub‑pipelines?