Kafka Streams 3.5.0 fetchAll API: missing pagination for large state stores
0 reputation · 05 Feb 2023, 11:31 UTC
Goal
Developers need a way to iterate over all key‑value pairs in a queryable state store while keeping memory usage bounded, especially when the store contains millions of entries.
Constraint
The fetchAll method introduced in Kafka Streams 3.5.0 returns a List containing the complete store contents, providing no offset‑based pagination or streaming alternative. Invoking fetchAll on a large store can exhaust the JVM heap and increase latency.
Uncertainty
It is unclear whether the Kafka Streams roadmap includes a paginated fetch API, and what the community‑endorsed pattern is for safely traversing large state stores without triggering out‑of‑memory errors.
Is there a planned enhancement to add paginated fetch methods to QueryableStore? If not, what architectural approach should be used to page through a state store safely? Are there any interim solutions, such as custom key range queries, that are officially supported?
apachekafka