Mocha retry configuration: missing flaky‑test threshold for retried passes
0 reputation · 20 Jun 2022, 18:13 UTC
0 reputation · 20 Jun 2022, 18:13 UTC
Define a policy that treats any test which succeeds only after one or more retries as flaky, so that CI can flag or quarantine it.
beforeEach/afterEach) also rerun per attempt, providing a natural reset point.this.currentRetry(), but there is no configuration option to mark a retried pass as unhealthy.35275 reputation · 21 Jun 2022, 01:07 UTC
No. Mocha (including 10.x) has no configuration option, CLI flag, or built-in reporter behavior that classifies a test passing on attempt 2+ as flaky. A retried pass is reported as an ordinary pass. The supported pattern is to detect it yourself in an afterEach hook using this.currentRetry(), collect the results, and surface them through a custom reporter or a CI artifact.
Two facts in your constraints combine into the detection rule:
this.currentRetry() returns the current attempt number (0 on the first attempt) and resets per test.afterEach runs after every attempt, and by then this.test.state is set for a passing attempt.So a test that passed on a retry is exactly: this.currentRetry() > 0 && this.test.state === 'passed'. That is your flaky threshold of "one or more retries" expressed as a single condition.
In a root-level setup file (loaded via --file or .mocharc):
// flaky-tracker.js
const flaky = [];
exports.flaky = flaky;
exports.mochaHooks = {
afterEach() {
if (this.currentRetry() > 0 && this.test.state === 'passed') {
flaky.push({
title: this.test.fullTitle(),
attempts: this.currentRetry() + 1,
});
}
},
};
Then a minimal custom reporter that fails or flags the build:
// flaky-reporter.js
const Base = require('mocha').reporters.Base;
const { flaky } = require('./flaky-tracker');
module.exports = class FlakyReporter extends Base {
constructor(runner) {
super(runner);
runner.on('end', () => {
if (flaky.length) {
console.log(`\n${flaky.length} flaky test(s):`);
for (const f of flaky) {
console.log(` ${f.title} (passed on attempt ${f.attempts})`);
}
// Quarantine mode: exit non-zero to fail CI
// process.exitCode = 1;
}
});
}
};
npx mocha --file ./flaky-tracker.js --reporter ./flaky-reporter.js
The retry API (this.retries(), this.currentRetry(), per-attempt hook re-execution) has been stable since roughly Mocha 5.x and is unchanged through 10.x, so this pattern is version-agnostic across that range. One caveat: arrow functions bind this lexically, so hooks must use function syntax to access the Mocha context — this has always been true, not a version difference.
this.retries(2) that fails on attempt 0 and passes on attempt 1 (e.g., a counter in module scope).attempts: 2.Because the whole test function re-runs per attempt, keep side effects idempotent and reset state in beforeEach. Also note the default spec reporter will never show flaky status on its own — the collected data only surfaces through your reporter or by writing the flaky array to a JSON artifact for CI to consume.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.