Question
RxJS expand pagination: choosing the right termination condition for a cursor-based API walk
Rahul Nova
0 reputation · 25 Feb 2025, 05:17 UTC
85.8K views
Loading…
0 reputation · 25 Feb 2025, 05:17 UTC
I'm building an RxJS pipeline that walks a cursor-paginated REST endpoint using the expand operator. Each page response returns { items, nextCursor }, and the projection fetches the next page whenever a cursor is present.
The unresolved decision is how the walk should terminate, because the endpoint's contract is ambiguous: some queries return an empty items array with a valid cursor, others return items with a missing cursor, and I also want a hard safety cap so a misbehaving endpoint cannot trigger unbounded requests against a rate-limited API.
The three candidate strategies seem to compose differently: returning EMPTY from the projection when the cursor is absent, adding takeWhile to stop on an empty page, or layering take(n) as an absolute page cap. I'm unsure how these interact when combined, and where catchError should sit so one failed page doesn't silently abort the entire walk.
Assuming RxJS 7 with top-level operator imports:
expand projection, downstream operators, or both, when empty pages and missing cursors mean different things?take(n) placed after expand reliably cap the number of HTTP requests, or can an in-flight projection still fire?catchError inside the projection the correct way to skip a failed page without ending the stream?29775 reputation · 25 Feb 2025, 15:48 UTC
To handle an ambiguous API contract while maintaining a safety cap, you should split your logic: termination logic belongs inside the expand projector, while safety guards belong in downstream operators.
The expand operator recurses as long as the projector returns an observable that emits. To stop the walk, the projector must return EMPTY. Because your API may return empty items arrays with valid cursors, you must rely on the nextCursor as the primary signal for continuation.
rxjsexpand(res => {
if (!res.nextCursor) return EMPTY;
return fetchPage(res.nextCursor).pipe(
catchError(err => {
console.error('Page failed', err);
return EMPTY; // Skip failed page and stop the walk
})
);
})To prevent unbounded requests from a misbehaving endpoint, use take(n) after expand. In RxJS, take(n) triggers an unsubscription once the limit is reached. Because expand manages its internal subscriptions, this unsubscription propagates upward and cancels any in-flight HTTP requests immediately.
The placement of catchError determines the survival of the stream:
EMPTY inside catchError, you effectively "swallow" the error for that specific page and terminate the recursive walk gracefully.Assuming RxJS 7, the recommended pipeline structure is:
initialRequest$.pipe(
expand(res => {
// Termination: Only continue if cursor exists
if (!res.nextCursor) return EMPTY;
return fetchPage(res.nextCursor).pipe(
// Error handling: Stop walk on failure without crashing
catchError(() => EMPTY)
);
}),
// Safety cap: Limits total pages emitted/requested
take(MAX_PAGES),
// Optional: Stop if a page is empty AND you've decided that's a termination signal
takeWhile(res => res.items.length > 0, true)
)This approach assumes that the first page is fetched via the seed observable. To verify this behavior, mock your API to return a repeating cursor; you should observe that take(MAX_PAGES) terminates the stream exactly at the limit, preventing an infinite loop.
One diagnostic detail needed: Does your API return a cursor on the final page that leads to a subsequent empty page? If so, takeWhile(res => res.items.length > 0, true) is required to avoid an extra unnecessary request.
Use comments to ask for clarification. Post a solution as an answer.
29,775 reputation · 25 Feb 2025, 08:49 UTC
When using expand to walk a cursor‑based API you must be sure that the recursion actually terminates. A common pitfall is a server that mistakenly returns the same cursor value on two consecutive pages. Because expand only stops when the projector emits EMPTY, the stream will keep requesting the same page forever.
Solution: keep a Set of seen cursors and return EMPTY if you encounter a repeat. This guard can live inside the projector, so it is part of the termination logic and does not interfere with downstream operators.
Example:
const seen = new Set();
source.pipe(
expand(res => {
if (!res.nextCursor || seen.has(res.nextCursor)) {
return EMPTY; // stop recursion
}
seen.add(res.nextCursor);
return fetchPage(res.nextCursor).pipe(
catchError(err => {
console.warn('Page failed', err);
return EMPTY; // skip this page but terminate the walk
})
);
}),
take(10), // hard safety cap
filter(Boolean) // drop the placeholder nulls if used
).subscribe(...);
take after expandPlacing take(10) downstream does *not* allow more than ten HTTP requests to be in flight. When the count is reached, take unsubscribes from the source, which propagates back to expand and cancels any pending HTTP calls immediately. No extra requests will start after the cap is hit.
If you want to skip a failed page but continue walking, returning EMPTY inside catchError will terminate that branch. Instead, emit a placeholder (e.g., of(null)) and filter it out downstream. This keeps the recursion alive while discarding the error page.