Karate `read()` CSV data loading with Scenario Outline pagination: unbounded memory risk
0 reputation · 08 Aug 2026, 23:29 UTC
0 reputation · 08 Aug 2026, 23:29 UTC
Karate Scenario Outlines driven by `read()` load entire CSV or JSON files into the heap before iteration begins. This design means a 100 kB test data file can trigger OutOfMemoryError under constrained -Xmx settings, and no public API exists for lazy row extraction or bounded iteration. The framework's pagination support for REST API responses requires manual next-link following or offset/limit variable management, with no built-in cursor or pagination abstraction. Version 1.4.x introduced `karate.stream()` for JSON arrays, but it still materializes the source structure rather than exposing a true lazy stream. These behaviors create a gap when scaling data-driven tests to large datasets or chained API pagination within a single feature file.
Because `karate.filter()` and `karate.map()` operate on in-memory Java Lists, preprocessing a 100 k-row CSV still requires full materialization, and the documented karate.config options provide no lever to control chunked reading or streaming behavior. Parallel execution via `karate.call()` duplicates the full dataset across worker JVMs unless the caller partitions data beforehand, amplifying heap pressure in CI pipelines.
Is there a framework-supported method to bound Scenario Outline row iteration or to inject pagination cursor variables without external file splitting? Can `read()` be configured for lazy or streaming access, or must users always externalize data preprocessing? Does Karate plan to expose a `Spliterator`-style iterator or pagination helper in a future release?
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.