Global retries versus suite-scoped retries for CI alert noise
0 reputation · 14 Dec 2020, 05:47 UTC
Preserving actionable CI alerts while reducing noise
The goal is to keep CI failure notifications useful by suppressing transient failures without masking real regressions. Cypress provides automatic test retries via the retries configuration and the this.retries() API, and the Cypress Dashboard provides flaky-test analytics from recorded historical runs.
The constraint is preserving actionable signal while reducing notification volume. Enabling retries can absorb intermittent failures and lower immediate alerts, but may hide underlying flakiness and delay root cause analysis. Disabling retries gives deterministic failure signals but increases alert volume from environmental or timing issues.
The unresolved decision is how to scope retry behavior. A global retry count suppresses noise uniformly but may hide regressions in critical paths, while suite-scoped retries allow different sensitivity per area.
Is a global retries setting compatible with maintaining distinct alert sensitivity for critical paths versus exploratory specs? Does Dashboard flaky-test analytics remain reliable when local retries are enabled? What precedence applies between config retries and this.retries() when both are defined for the same suite?