Flaky test not reproduced by RSpec --bisect: diagnosing non‑order‑dependent failures

The direct answer

If --bisect fails to reproduce the failure, the most likely explanation is that the flake is not order-dependent. --bisect works by permuting example order under a pinned seed to find a minimal reproducing set — it can only isolate failures that deterministically appear for some ordering. A failure that appears and disappears under the same seed is, by definition, outside what bisect can find. One caveat before concluding anything: a passing bisect run does not prove order-independence; it only shows the failure wasn't reproduced under the orders bisect explored. Very slow suites also make bisect impractical even for genuinely order-dependent bugs.

Indicators the failure is not order-dependent

Look at the failure message and context first. Non-order-dependent flakes typically show:

• Timeouts, stale-element errors, or assertions that race against asynchronous work (Capybara waiting, background jobs, Ajax).
• Sensitivity to time-of-day, dates, timezones, or daylight-saving boundaries.
• Random data collisions — e.g., Faker or factory-generated values violating a uniqueness constraint only occasionally.
• Dependence on the network, filesystem, external APIs, or shared CI resources.
• Failures that only occur under a parallel runner (e.g., parallel_tests), where cross-process contention on the database or ports mimics order dependence.

The key experiment: repetition under a fixed order

The decisive test is to run the failing example alone, with the recorded seed, many times. Identical order plus intermittent failure means ordering is not the variable:

for i in $(seq 1 100); do
  bundle exec rspec --seed 12345 spec/models/thing_spec.rb \
    || echo "FAILED on iteration $i"
done

Interpretation:

1. Fails in isolation, intermittently — the cause is internal to the example or its setup: timing, concurrency, randomness, or an external dependency. Bisect was never going to help.
2. Always passes in isolation, fails in the full suite — now order-dependence is plausible, and bisect's failure to reproduce may just mean it didn't explore the triggering order. Re-run the full suite under the original failing seed several times to confirm the failure is seed-stable.

Differentiating timing/concurrency from state leakage

Once you have a stochastic single-example failure, narrow the class with controlled elimination:

• Freeze time with Timecop and re-run the loop. If the flake disappears, it's time-dependent.
• Stub external HTTP with VCR or WebMock. If the flake disappears, an external service is the culprit.
• Log the randomness: temporarily record generated values, timestamps, thread IDs, and database row counts at the failure point, then diff failing vs. passing runs.
• Reproduce serially if you use a parallel runner, before attributing anything to ordering.
• Audit for leaked state as a complement to bisect: class variables, memoized globals, mutations to Rails.application.config, ENV changes, or an incomplete DatabaseCleaner/transactional-fixtures setup.

Complementary tooling

rspec --only-failures (with a persisted status file) and rspec-retry help quantify flake rate and triage, but treat retry as a measurement tool, not a fix — it masks root causes. RSpec has no built-in timing-flake detector; the loop-plus-elimination approach above is the standard substitute.

Uncertainty and one missing detail

This guidance assumes a reasonably current RSpec 3.x and a suite that runs non-interactively. The one detail that would change the recommendation: does the failing example pass when run alone? If yes, invest in seed-stable full-suite reruns and state-leak auditing; if no, skip bisect entirely and go straight to the timing/external-dependency elimination steps.