Cypress Cloud Slack and GitHub alerts: failure-only notifications without losing flaky test visibility
0 reputation · 25 Jun 2023, 16:53 UTC
0 reputation · 25 Jun 2023, 16:53 UTC
Configure Cypress Cloud notifications so the team is alerted to genuine run failures in Slack and GitHub, without a message for every passing run, while still keeping flaky tests visible somewhere for triage.
Alerting is configured at the organization or project level in Cypress Cloud, not per spec file, so any filter applies broadly. The GitHub integration posts commit status checks and PR comments, and the Slack integration can be scoped by run outcome and branch. The concern is that restricting Slack to failed runs on main and release branches may also silence signal from feature branches where regressions are first introduced. Separately, flaky test detection flags tests that pass and fail across runs without code changes, but it is unclear whether flake analytics remain useful if flaky specs are muted, and whether muting risks hiding real failures.
Some alerting and flake-detection capabilities appear to depend on the Cloud plan tier, so the exact filter options available may differ from what documentation describes.
29775 reputation · 25 Jun 2023, 18:53 UTC
Yes, you can get failure-only alerting without losing flake visibility, but the pieces live in different places. Slack and GitHub notifications are configured per integration at the organization or project level, so filters apply broadly, not per spec. Flake analytics are derived from recorded run history, so muting notifications does not remove flaky specs from analytics — only skipping or excluding those specs from CI does. The clean split is: failure-only Slack scoped to main and release branches, and GitHub statuses plus PR comments on feature branches as the early-warning channel.
The Slack integration exposes both event selection (failing runs, passing runs, and similar) and branch selection, and combining them in one integration is the intended pattern — for example, "failed runs on main and release/* only." The exact set of combinable options is UI- and plan-dependent, so confirm against your own project's integration settings rather than assuming the documentation matches your tier.
If feature-branch failures must also reach Slack, don't widen the main integration. Add a second Slack integration pointing at a separate channel or webhook scoped to feature branches, so the low-noise channel stays low-noise and the noisy channel is opt-in.
Muting a known-flaky spec affects notifications, not data. Flake detection is computed from recorded run history — the same spec passing and failing across runs without code changes — and that history is not deleted when you mute. Flake analytics generally survive muting.
What does degrade flake signal is anything that stops the spec from running: skipping it, excluding it from CI, or trimming feature-branch runs. Fewer recorded runs means a smaller sample for detection to learn from.
One caution: muting is triage hygiene, not a fix. A muted spec that fails for a genuine reason still produces a failing run and, depending on configuration, a failing GitHub status check. The failure signal isn't gone — it's just quieter.
Commit status checks and pull-request comments are separate behaviors of the GitHub integration, and you control whether each is posted and for which branches. Status checks are the pass/fail signal that branch protection consumes, so be careful here: if you suppress passing-run statuses or scope statuses too narrowly, a branch protection rule that requires the Cypress check on every commit will block merges because the expected status never arrives. Keep status checks reporting on every relevant commit; reduce noise with PR comment settings instead.
One diagnostic question that changes everything: are feature-branch runs recorded to Cypress Cloud at all? If they aren't, no alerting or flake configuration can produce signal from them, and the fix is in your CI workflow, not in notification settings.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.