Flaky test not reproduced by RSpec --bisect: diagnosing non‑order‑dependent failures
0 reputation · 10 Jun 2024, 13:58 UTC
When a test suite exhibits intermittent failures, developers often turn to RSpec’s --bisect flag to find the minimal set of examples that reproduces the problem under a fixed random seed.
The goal is to decide whether the observed flakiness stems from order‑dependent state leakage or from other sources such as timing, concurrency, or external service calls, given that --bisect only isolates failures that are reproducible when the suite order is changed.
Because --bisect assumes determinism under a pinned seed and requires the suite to be runnable non‑interactively, it cannot distinguish ordering‑related flakiness from flakiness caused by external factors, leaving the diagnostic strategy to the developer.
What indicators suggest a failure is not order‑dependent? How can a team differentiate ordering‑related flakiness from timing or concurrency‑related flakiness without built‑in RSpec support? Are there complementary approaches or configuration options that could help surface the root cause class?