Repeated health-check failures trigger alert storms in Dropwizard admin endpoint
0 reputation · 08 Jul 2020, 20:06 UTC
Goal
Determine whether Dropwizard should provide a built-in alerting dispatcher that aggregates metric thresholds and health-check results into a single notification channel, thereby reducing noise while preserving visibility of genuine issues.
Constraints and uncertainty
The dispatcher must avoid adding noticeable runtime overhead, must not suppress legitimate alerts, and must remain optional to preserve backward compatibility with existing external alerting integrations (e.g., Prometheus Alertmanager, PagerDuty).
It is unclear how such a dispatcher would interact with the current Metrics reporters and HealthCheck endpoint without causing duplicate notifications or increasing load on the admin interface.
Specific questions:
- What are the trade-offs of integrating a threshold-based alerting dispatcher into Dropwizard’s core?
- How would the dispatcher coordinate with existing ConsoleReporter, Slf4jReporter and health-check polling to prevent duplicate or missed alerts?
- What minimal configuration would enable users to activate or deactivate the dispatcher while keeping current external integrations functional?