Can git bisect run distinguish a true regression from a flaky test failure?

Git bisect run has no built-in mechanism to detect flaky failures

git bisect run interprets exit codes literally: 0 means good, anything else means bad. It cannot distinguish between a genuine regression and a transient environment issue like a flaky test, network timeout, or resource contention.

Confirmed behavior

• The bisect algorithm performs a deterministic binary search based solely on the pass/fail signal you provide.
• If your test script exits non-zero due to a flaky test, Git records that commit as bad, regardless of whether the failure is reproducible.
• Git offers no recovery mechanism for commits misclassified by transient errors.

What the test script must do

The script is responsible for filtering noise before returning an exit code to Git. Common strategies include:

1. Retry logic with consensus: Run the test suite multiple times and return non-zero only if all runs fail.
2. Failure pattern detection: Check for specific error signatures (e.g., timeout messages) and return 0 if the failure matches known flaky patterns.
3. Probabilistic thresholds: Define a minimum failure rate (e.g., 3 of 5 runs) before marking as bad.

Practical implementation

#!/bin/bash # Example: retry-based flaky-test filter MAX_RETRIES=3 SUCCESS_COUNT=0 for i in $(seq 1 $MAX_RETRIES); do if ./run-tests.sh; then ((SUCCESS_COUNT++)); fi # Early exit if any run succeeds [ $SUCCESS_COUNT -gt 0 ] && break done # Return 0 only if all runs passed test $SUCCESS_COUNT -eq $MAX_RETRIES

Verification and recovery

Once bisect completes, use git bisect log to review the commit sequence. If you suspect flaky misclassifications, manually verify the identified bad commit and use git bisect replay with a corrected log to exclude suspected false positives.

Key limitation

Adding retries increases bisect time linearly with the number of retries. There is no way to make Git itself smarter about intermittent failures—the intelligence must be in your test harness.