Kafka Streams 3.5.0 fetchAll API: missing pagination for large state stores

Answer to the question

The fetchAll method introduced in Kafka Streams 3.5.0 returns a List that contains the entire contents of a queryable state store. It has no built‑in pagination or streaming interface. There is no publicly documented roadmap entry for a paginated fetch API in the current 3.5.x series, and the official Javadoc for QueryableStore and KeyValueStore makes no mention of such an enhancement.

Confirmed facts

• Calling fetchAll() on a store with millions of entries can exhaust the JVM heap and increase latency.
• The KeyValueStore interface provides an all() method that returns a KeyValueIterator, which streams results lazily.
• A KeyValueIterator can be processed in batches and must be closed after use to avoid resource leaks.
• Key‑range queries are not directly supported for arbitrary key types; only comparable key types can be filtered manually.

Recommended architectural approach

Use the all() iterator to read the store sequentially, and wrap it in a simple pagination helper that returns a fixed‑size batch on each call. This keeps memory usage bounded and avoids OOM.

public class PaginatedStoreReader<K, V> {
    private final KeyValueIterator<K, V> iterator;
    private final int batchSize;

    public PaginatedStoreReader(KeyValueStore<K, V> store, int batchSize) {
        this.iterator = store.all();
        this.batchSize = batchSize;
    }

    public List<KeyValue<K, V>> nextBatch() {
        List<KeyValue<K, V>> batch = new ArrayList<>(batchSize);
        int count = 0;
        while (iterator.hasNext() && count < batchSize) {
            batch.add(iterator.next());
            count++;
        }
        return batch;
    }

    public void close() { iterator.close(); }
}

Typical usage:

PaginatedStoreReader<String, Integer> reader =
        new PaginatedStoreReader<>(store, 10_000);
try {
    List<KeyValue<String, Integer>> batch;
    while (!(batch = reader.nextBatch()).isEmpty()) {
        // process batch
    }
} finally {
    reader.close();
}

Why this works

• The iterator streams entries one by one, so only the current batch is held in memory.
• Because the iterator is backed by RocksDB (or another local storage), the read is efficient and does not load the entire store into heap.
• Closing the iterator releases the underlying file handles and avoids leaks.

Interim solutions and alternatives

• Key‑range filtering: If your keys are comparable (e.g., String or Long), you can open the iterator and skip entries until a desired start key, then read a limited number of entries. This is linear in the offset and not efficient for deep pagination.
• Custom index store: For frequent random page access, maintain a lightweight auxiliary store that holds the keys in sorted order and their offsets. Use that index to seek into the main store.
• External export: Periodically export the store to an external system (e.g., Kafka topic, HDFS) and use that as the source for pagination.

When to consider a roadmap change

At this time, the Kafka community has not opened an issue to add a paginated fetch API to QueryableStore. If your use case requires deep pagination or random page access, consider filing an issue on the Apache Kafka issue tracker and provide a concrete use case. Until then, the iterator‑based approach is the recommended pattern.

Diagnostic question

Do you need to support random page jumps (e.g., page 10 of 100) or only forward sequential reads? This determines whether a simple iterator wrapper suffices or you need an auxiliary index.