Using JUnit 5's @RepeatedTest to Catch Flaky Tests

Problem: Flaky tests hide in CI

Intermittent failures are hard to spot when a test runs only once per build. A race condition, a timing issue, or a resource leak may surface only occasionally, causing the CI pipeline to pass most of the time and fail sporadically. Teams often waste time reproducing the failure locally or ignore it as "noise".

Thesis: @RepeatedTest surfaces intermittent problems by executing the same test many times in a single run

JUnit 5 provides the @RepeatedTest annotation. When you place it on a test method, the test runner executes that method the specified number of times, reporting each iteration as a separate test node. This makes occasional failures visible immediately, because a single bad iteration will cause the overall test to fail.

How @RepeatedTest works

The annotation lives in org.junit.jupiter.api.RepeatedTest and requires JUnit Jupiter 5.0 or newer. Its value attribute defines the repetition count, for example @RepeatedTest(10). Each iteration gets its own display name built from the method name and the current index, but you can override it with the name attribute using placeholders such as {displayName}, {currentRepetition}, and {totalRepetitions}.

Lifecycle methods behave as follows:

• @BeforeAll and @AfterAll run once for the whole repeated‑test group.
• @BeforeEach and @AfterEach run before and after every single repetition.
• Any extensions (e.g., @TempDir, MockitoExtension) are applied to each iteration, giving each run a fresh context.

Worked example

Suppose you have a service that caches the result of an expensive calculation. You want to verify that the cache does not accidentally return a stale value under concurrent access. The following test repeats the scenario five times.

import org.junit.jupiter.api.*;

import java.util.concurrent.*;

import static org.junit.jupiter.api.Assertions.*;

class CacheServiceTest {

    private CacheService service;

    @BeforeEach
    void setUp() {
        // This runs before each repetition
        service = new CacheService();
    }

    @RepeatedTest(value = 5, name = "Repetition {currentRepetition} of {totalRepetitions}")
    void testCacheUnderConcurrentLoad(RepetitionInfo repetitionInfo) throws Exception {
        ExecutorService exec = Executors.newFixedThreadPool(10);
        Callable task = () -> service.computeIfAbsent("key", () -> {
            // Simulate expensive work
            try {
                Thread.sleep(10);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                throw new RuntimeException(e);
            }
            return 42;
        });

        Future[] futures = new Future[10];
        for (int i = 0; i < futures.length; i++) {
            futures[i] = exec.submit(task);
        }

        for (Future f : futures) {
            assertEquals(42, f.get(), "All threads should see the same cached value");
        }
        exec.shutdownNow();
    }
}

When you execute this test with Maven (mvn test) or Gradle (./gradlew test), you will see five test nodes in the IDE’s test tree, each labelled "Repetition 1 of 5", "Repetition 2 of 5", and so on. The console output (or the test report) shows that setUp() is invoked five times, while a hypothetical @BeforeAll method would appear only once.

Trade‑off and limitation

The main cost is that any expensive setup placed in @BeforeEach (or @AfterEach) is multiplied by the repetition count. If creating the test fixture takes seconds, a @RepeatedTest(50) could add minutes to your suite. In such cases consider moving the heavy work to @BeforeAll or using a shared resource managed by an extension, and reserve @BeforeEach for lightweight state that must be isolated per iteration.

Another limitation is that state intended to persist across repetitions must be stored in a field initialized in @BeforeAll (or a static variable). If you mistakenly keep it in an instance field and re‑initialize it in @BeforeEach, each repetition starts from a clean slate, which can hide bugs that depend on accumulated state (e.g., a leak that only appears after many iterations).

Actionable advice

1. Identify tests that are suspected of being flaky or that exercise concurrent code.
2. Add @RepeatedTest with a modest count (e.g., 5‑10) to see whether failures appear consistently.
3. Inspect the test report: if any iteration fails, investigate the specific repetition; the RepetitionInfo parameter gives you the current index if you need it in logic.
4. Profile the setup cost. If @BeforeEach is expensive, refactor to use @BeforeAll or a scoped extension.
5. Use the name attribute to make the test output readable in CI logs, which helps pinpoint the problematic iteration.

By treating @RepeatedTest as a diagnostic tool rather than a permanent fixture, you gain confidence that intermittent issues are caught early without inflating your test suite unnecessarily.