Architecting Data-Driven Suites with JUnit 5 Dynamic Tests

The Problem: Static Test Definitions vs. Dynamic Data

Standard JUnit tests and @ParameterizedTest require the number of test cases to be known or defined by a static source at the time the test class is instantiated. When test cases must be generated based on a runtime discovery process—such as scanning a directory for JSON payloads, querying a database for active configuration sets, or fetching a list of endpoints from a service registry—static declarations become a bottleneck. The takeaway is that @TestFactory allows you to shift test case definition from compile-time to runtime, enabling a truly data-driven architecture.

Requirements for Dynamic Generation

To implement dynamic tests, the following architectural constraints must be met:

• Return Types: The method must be annotated with @TestFactory and return a Stream<DynamicTest>, Collection<DynamicTest>, or Iterable<DynamicTest>.
• Execution Timing: Tests are generated during the test discovery phase and executed sequentially.
• Lifecycle Restrictions: Because dynamic tests are not individual @Test methods, they do not trigger @BeforeEach or @AfterEach for every generated case. They only respect the lifecycle of the factory method itself.

Smallest Suitable Design

The most efficient implementation uses a Stream to lazily generate tests, preventing the entire dataset from loading into memory before execution starts. This is particularly critical when dealing with large external files.

import org.junit.jupiter.api.*;
import org.junit.jupiter.api.DynamicTest;
import java.util.stream.Stream;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;
import static org.junit.jupiter.api.Assertions.*;

class DataDrivenArchitectureTest {

    @TestFactory
    Stream<DynamicTest> generateTestsFromExternalSource() {
        // Assume getTestData() fetches a list of inputs from a file or API
        return TestDataSource.getTestData().stream()
            .map(data -> dynamicTest(
                "Testing scenario: " + data.getScenarioName(),
                () -> {
                    var result = service.process(data.getInput());
                    assertEquals(data.getExpected(), result);
                }
            ));
    }
}

Trust and Data Boundaries

When generating tests from external sources, you introduce a trust boundary between the test runner and the data source. Consider these boundaries:

• Input Validation: If the data source is a CSV or JSON file, validate the schema before passing it to dynamicTest. A malformed file should cause the @TestFactory method to fail early rather than producing hundreds of broken tests.
• Resource Isolation: Since @BeforeEach is not available for individual dynamic tests, any required state (like a database transaction or a temporary folder) must be managed manually within the Executable lambda or via a wrapper object.
• Timeout Boundaries: External data can lead to an unbounded number of tests. Implement a .limit(n) on your stream to prevent CI/CD pipelines from hanging due to an unexpectedly large data source.

Operational Checks and Verification

To verify that dynamic tests are correctly integrated into your pipeline, perform the following checks:

1. IDE Inspection: Run the test in your IDE. Verify that the test runner displays a nested tree structure where the @TestFactory method acts as a container and each DynamicTest appears as a child with its unique display name.
2. CLI Discovery: Run the tests using the JUnit Platform Console Launcher to ensure discovery works without IDE assistance:

   java -jar junit-platform-console-standalone.jar --select-class com.example.DataDrivenArchitectureTest

   Risk: Ensure the standalone JAR version matches your JUnit 5 runtime version to avoid discovery failures.
3. Result Mapping: Intentionally break one data record in your source file. Verify that only that specific dynamic test fails while others continue to execute.

Failure Modes and Design Shifts

Dynamic tests fail differently than static tests. If the @TestFactory method itself throws an exception (e.g., FileNotFoundException), no tests are generated, and the entire suite is marked as failed.

Conditions that would change this design:

• Need for Parallelism: Dynamic tests are executed sequentially. If your data-driven suite requires parallel execution to meet performance SLAs, you must shift back to @ParameterizedTest with @Execution(ExecutionMode.CONCURRENT).
• Complex Setup/Teardown: If every single data-driven case requires a fresh database migration or a heavy container restart, the lack of @BeforeEach support makes @TestFactory impractical.