Does Karate evaluate the entire JSON array passed to 'call' before iteration when paginating large datasets?
0 reputation · 03 Nov 2024, 13:44 UTC
0 reputation · 03 Nov 2024, 13:44 UTC
To paginate a large dataset in Karate, testers often want to drive the 'call' keyword with a JSON array that contains page‑size and offset values, avoiding the need to maintain a bulky Examples table.
While the documentation warns that large Examples tables can increase memory consumption during execution, it does not specify whether passing a JSON array to 'call' behaves the same way or if the array is evaluated lazily by the embedded JavaScript engine.
Does Karate load the whole JSON array into memory before starting the call loop? If it does, what patterns can be used to stream pages incrementally? Is there a configuration or alternative keyword that enables lazy evaluation for large data-driven calls?
33275 reputation · 03 Nov 2024, 19:34 UTC
Karate evaluates the entire JSON array that is passed to a call loop before any iteration starts. The array is materialised by the embedded JavaScript engine (GraalVM or Nashorn, depending on the version) and occupies heap space for the duration of the call.
Because the array is fully loaded, memory usage grows linearly with the number of elements. This mirrors the behaviour of large Examples tables and can lead to OutOfMemoryError if the array is very large.
To avoid loading all pages at once, use a recursive feature call that fetches one page at a time. Each invocation passes the next offset and stops when the response indicates no more data.
# pagination.feature
Scenario: Process Pages
* def offset = 0
* def limit = 100
* call read('pagination.feature') { offset: offset, limit: limit }
Scenario: Page Worker
Given url 'https://api.example.com/data'
And param offset = offset
And param limit = limit
When method get
Then status 200
* def hasMore = response.length == limit
* if (hasMore) call read('pagination.feature') { offset: offset + limit, limit: limit }
Run the feature while monitoring the JVM heap (e.g., with JVisualVM). Observe the spike when a large array is used vs. the steady profile of the recursive approach.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.