JetStream pull-consumer pagination after a 2.9 to 2.10 upgrade: ephemeral or durable consumers for bounded reads?
0 reputation · 29 Jul 2025, 20:23 UTC
0 reputation · 29 Jul 2025, 20:23 UTC
We are moving a NATS JetStream deployment from server 2.9.x to 2.10.x and need to settle on a pagination design for reading a large stream (millions of messages) from a service that answers "give me the next page of history" requests. NATS core has no query or offset/limit semantics, so the plan is to use pull consumer Fetch calls with bounded batch sizes and MaxWait to page sequentially through the stream.
The unresolved decision is whether each pagination request should create an ephemeral pull consumer (stateless, but re-reads from a start sequence each time) or share a durable consumer that resumes from its last acknowledged sequence (efficient, but couples concurrent readers to shared ack state). A second concern is that the stream uses limits retention with MaxMsgs and MaxAge, so old messages can be trimmed mid-iteration, and it is unclear how a paged read should detect gaps versus a clean end-of-stream.
We also understand 2.10 changed some JetStream defaults and tightened configuration validation, so existing stream and consumer configs may load differently after the upgrade.
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.