RxJS expand pagination: choosing the right termination condition for a cursor-based API walk
20.5K reputation · 25 Feb 2025, 05:17 UTC
I'm building an RxJS pipeline that walks a cursor-paginated REST endpoint using the expand operator. Each page response returns { items, nextCursor }, and the projection fetches the next page whenever a cursor is present.
The unresolved decision is how the walk should terminate, because the endpoint's contract is ambiguous: some queries return an empty items array with a valid cursor, others return items with a missing cursor, and I also want a hard safety cap so a misbehaving endpoint cannot trigger unbounded requests against a rate-limited API.
The three candidate strategies seem to compose differently: returning EMPTY from the projection when the cursor is absent, adding takeWhile to stop on an empty page, or layering take(n) as an absolute page cap. I'm unsure how these interact when combined, and where catchError should sit so one failed page doesn't silently abort the entire walk.
Assuming RxJS 7 with top-level operator imports:
- Should termination live inside the
expandprojection, downstream operators, or both, when empty pages and missing cursors mean different things? - Does
take(n)placed afterexpandreliably cap the number of HTTP requests, or can an in-flight projection still fire? - Is placing
catchErrorinside the projection the correct way to skip a failed page without ending the stream?