TimeoutExtension outcome classification: failure vs error for timed-out tests
0 reputation · 28 Apr 2021, 03:09 UTC
JUnit 5's TimeoutExtension currently reports a test that exceeds its @Timeout limit as a failure by throwing org.opentest4j.TestTimedOutException. The extension uses a watchdog thread that interrupts the test thread; if the test code does not respond to interruption, the extension still marks the test as failed rather than treating it as an error or a distinct cancellation state.
This behavior contrasts with Assumptions.assumeTrue(false), which causes the test to be reported as ABORTED, demonstrating a separate cancellation path. Community discussion has not settled whether a timed-out test should be classified as a failure, an error, or a new outcome type, nor whether extensions can customize the reported outcome beyond the default exception.
Constraints include the reliance on thread interruption for cancellation, which can interfere with tests that intentionally use interruption for their own logic, and the risk that swallowing InterruptedException prevents the extension from detecting the timeout at all.
- Should a test that exceeds its timeout be reported as a failure, an error, or a distinct outcome category?
- Can an extension or listener override the default
TestTimedOutExceptionto produce a custom test status without modifying the core extension?