Choosing a Data-Driven Testing Pattern in Karate Labs v1.x

The Decision: How to Feed Test Data into Karate Scenarios

Karate Labs v1.x offers three distinct patterns for data-driven testing. Each handles data volume, reporting granularity, parallel execution, and maintenance differently. The right choice depends on dataset size, whether data is static or dynamic, and how much per-row visibility you need in CI reports.

Constraints That Shape the Choice

• Dataset size: Under 50 rows favors inline tables; 50–10,000 rows favors external files; beyond 10,000 rows needs streaming or partitioning.
• Data source: Static CSV/JSON files, property files, or runtime sources (database, API).
• Reporting needs: Per-row pass/fail in JUnit XML and HTML reports vs. single-scenario roll-up.
• Parallel execution: Native thread distribution across rows vs. manual partitioning.
• Team workflow: Non-technical editors updating data vs. developers managing code.

Pattern Comparison

Pattern	Best For	Reporting Granularity	Parallel Support	Maintenance	Type Handling
Scenario Outline + Examples	Small static datasets (<50 rows), readable Gherkin	One reported Scenario per row; fine-grained pass/fail in CI	Native: rows distributed across threads via karate.parallel()	Data inline in feature file; bloats beyond ~200 rows	All values strings; coerce with __arg + 0 or karate.toJava()
karate.call() / karate.repeat() over JSON/CSV	Large/external datasets (50–10k rows), version-controlled data files	Single Scenario with internal iterations; failure marks whole Scenario failed unless wrapped in karate.abort()	Sequential inside one thread; manual partition + karate.fork() for parallelism	External files decouple data from logic; non-technical editors can update CSV	Native JSON/CSV types preserved automatically
Dynamic Scenario generation (karate.setup() returning [{name, fn}])	Truly dynamic sources (DB queries, API calls at runtime)	Generated scenarios appear as separate entries but lose native retry/flaky handling	Parallel works but order non-deterministic; no row-level control	Highest complexity; logic and data intertwined in JS; harder to debug	Full JS type control; runtime coercion as needed

Trade-offs in Detail

Scenario Outline: Simplicity with Reporting Clarity

Scenario Outline is the Gherkin-native approach. Each row in the Examples table becomes a distinct Scenario in reports. This gives you per-row pass/fail in JUnit XML and the Karate HTML report without extra code. Parallel execution (mvn test -Dkarate.threads=4) distributes rows across threads automatically.

Constraints: Examples values must be literals or come from karate.properties['key'] — you cannot reference variables defined in Background. All values are strings; numeric operations require coercion (* def num = __arg + 0). Large tables (>200 rows) bloat feature files and slow IDE parsing.

Loop Over External Data: Scale and Separation

Reading CSV or JSON via read('classpath:data.csv') (auto-converts CSV to list of maps) or read('data.json') keeps data out of feature files. Use karate.repeat() for tight inline loops — it avoids the overhead of karate.call() re-evaluating a feature per iteration.

Reporting caveat: the whole loop runs as one Scenario. A single iteration failure fails the Scenario. For per-iteration visibility, wrap each iteration in a try/catch and call karate.info() or karate.attach(). Parallelism requires manual data partitioning and karate.fork().

For files >10,000 rows, avoid loading the entire array into heap. Stream via Java interop or split into smaller files.

Dynamic Generation: Maximum Flexibility, Maximum Cost

karate.setup() can return an array of {name: 'scenario name', fn: function() { ... }} objects built from runtime data (database rows, API responses). This is the only pattern that handles truly dynamic sources.

Costs: loses Gherkin readability, complicates debugging (stack traces point to generated functions), and disables Karate's native retry/flaky-test handling for generated scenarios. Use only when data cannot be known at feature-load time.

Concrete Implementation: Same Logic, Three Patterns

Below are minimal feature files exercising a mock endpoint /api/users/{id}. Each pattern validates that the response id matches the input.

1. Scenario Outline (outline.feature)

Feature: Data-driven via Scenario Outline

Background:
  * url 'http://localhost:8080'
  * def expectedStatus = 200

Scenario Outline: Get user by id
  Given path 'api', 'users', __arg.id
  When method get
  Then status expectedStatus
  And match response.id == __arg.id

Examples:
  | id |
  | 1  |
  | 2  |
  | 3  |

Run: mvn test -Dkarate.options='--tags @outline'. HTML report shows three separate Scenario entries. Parallel: mvn test -Dkarate.threads=2 distributes rows across two threads.

2. Loop Over CSV (loop.feature)

Feature: Data-driven via CSV loop

Background:
  * url 'http://localhost:8080'
  * def userData = read('classpath:users.csv')

Scenario: Loop over users from CSV
  * karate.repeat(userData, function(row) {
    Given path 'api', 'users', row.id
    When method get
    Then status 200
    And match response.id == row.id
  })

CSV (src/test/resources/users.csv):

id,name
1,Alice
2,Bob
3,Carol

Note: CSV headers with spaces become map keys with spaces — access via row['header name'], not dot notation. Types preserved: row.id is a number.

Run: mvn test -Dkarate.options='--tags @loop'. Single Scenario in report. For per-row visibility, add karate.info('User ' + row.id) inside the loop.

3. Dynamic Generation (dynamic.feature)

Feature: Data-driven via dynamic scenarios

Background:
  * url 'http://localhost:8080'

@dynamic
Scenario: Generate scenarios from runtime source
  * def dynamicScenarios = function() {
    var list = [];
    // Simulate runtime data fetch (e.g., DB query)
    var ids = [10, 20, 30];
    for (var i = 0; i < ids.length; i++) {
      var id = ids[i];
      list.push({
        name: 'Dynamic user ' + id,
        fn: function() {
          Given path 'api', 'users', id
          When method get
          Then status 200
          And match response.id == id
        }
      });
    }
    return list;
  }
  * configure setup = dynamicScenarios

Run: mvn test -Dkarate.options='--tags @dynamic'. Three generated scenarios appear in report. No native retry support.

Validation Checklist

1. Create minimal project: mvn archetype:generate -DarchetypeGroupId=com.intuit.karate -DarchetypeArtifactId=karate-archetype -DarchetypeVersion=1.4.0
2. Add three features as above, plus a mock: * def mock = karate.mockService() in a mock.feature started via karate.callSingle().
3. Run with tags: mvn test -Dkarate.options='--tags @datadriven' (tag each feature). Compare target/karate-reports for row granularity, pass/fail counts, execution time.
4. Enable parallel: mvn test -Dkarate.threads=4. Verify thread distribution in report timeline view — Scenario Outline rows should spread across threads; loops stay in one thread unless forked.
5. Test data volumes: Create CSV/JSON with 1, 100, 10,000 rows. Observe startup memory and parse time. Scenario Outline bloats at ~200 rows; loops handle 10k but heap grows; dynamic generation scales but adds setup latency.
6. Check JUnit XML: target/surefire-reports/*.xml. Scenario Outline produces one <testcase> per row; loops produce one <testcase>; dynamic produces one per generated scenario.

Limitations and When to Re-evaluate

• Scenario Outline cannot use Background variables in Examples — if you need computed values, switch to loops or dynamic generation.
• Parallel execution with Scenario Outline may reorder rows; never rely on execution order for stateful tests.
• Dynamic generation disables Karate's retry mechanism — flaky tests in generated scenarios won't auto-retry.
• CSV headers with special characters require bracket notation (map['header name']).

Decision Summary

Start with Scenario Outline for static data under 50 rows — it gives the best reporting and parallelism with least code. Move to CSV/JSON loops when data grows, changes frequently, or is maintained by non-developers. Reserve dynamic generation for cases where the dataset truly cannot be known until runtime (e.g., querying a test database for current user IDs). Validate your choice by running the three patterns against a mock and comparing the HTML and JUnit outputs — the report structure will confirm whether per-row visibility meets your CI requirements.