Repeated health-check failures trigger alert storms in Dropwizard admin endpoint

Answer

Dropwizard can expose a lightweight, optional AlertingDispatcher that listens to health‑check state changes and metric threshold breaches, aggregates them, and forwards a single notification to a configured channel. This keeps the core API unchanged, adds negligible runtime overhead, and preserves existing external integrations.

Trade‑offs of a Core Dispatcher

• Pros
  • Reduces alert storms by deduplicating failures that would otherwise hit each reporter separately.
  • Centralises threshold logic, making it easier to tune and audit.
  • Optional flag keeps backward compatibility for projects that prefer external alerting systems.
• Cons
  • Introduces a new failure domain; if the dispatcher crashes, alerts may be lost unless a fallback is in place.
  • Potentially increases the memory footprint by holding state for each health‑check and metric.
  • Requires careful design to avoid duplicate notifications when multiple reporters are active.

Co‑ordination with Existing Reporters

1. Event Subscription: The dispatcher registers as a listener on the HealthCheckRegistry and on each Metric that has a threshold configured. Dropwizard’s reporter APIs already expose addListener hooks, so no polling loop is needed.
2. Deduplication: Each event is keyed by source (health‑check name or metric name) and severity. The dispatcher keeps the last sent state and only emits when the state changes or a configurable cool‑down expires.
3. Fallback Paths: When the dispatcher is disabled, events flow directly to the configured reporters (ConsoleReporter, Slf4jReporter, PrometheusReporter, etc.) as before. The dispatcher can also expose its own reporter that simply forwards aggregated events to a chosen sink.
4. Metrics Export: The dispatcher does not alter the metrics data; it merely reads the values. Therefore Prometheus scrapes and other exporters remain fully functional.

Minimal Configuration

Dropwizard’s application.yml can enable or disable the dispatcher with a single flag and optional channel settings:

# application.yml
alerting:
  enabled: false            # set to true to activate the dispatcher
  channel: "slack"           # optional: "slack", "pagerduty", "email", etc.
  slack:
    webhookUrl: "https://hooks.slack.com/services/..."
  pagerduty:
    integrationKey: "..."
  thresholds:
    # metricName: thresholdValue
    http.requests: 2000
    db.connections: 50

When enabled is false, Dropwizard behaves exactly as before. Setting it to true activates the dispatcher; the channel configuration is optional – if omitted, the dispatcher falls back to writing aggregated events to the console (or a configured Slf4jReporter).

Missing Diagnostic Detail

To fine‑tune the recommendation, could you confirm whether you already use a dedicated alerting platform (e.g., Prometheus Alertmanager, PagerDuty) that consumes Dropwizard metrics, or do you rely on Dropwizard’s reporters for all alerting? This determines whether the dispatcher should primarily deduplicate internal alerts or act as a bridge to an external system.