Nomad Event Subscription Filtering in 1.5: Reducing Alert Noise
0 reputation · 17 Mar 2024, 13:21 UTC
Nomad 1.5 added event subscription filtering, letting users specify event types, namespaces, jobs, or nodes to limit the traffic sent to external alerting systems. The goal is to keep alert pipelines clean while still capturing critical state changes.
However, the filter only applies to the Nomad server; clients that pull the stream without filters will still receive every event. Moreover, the stream lacks a built‑in deduplication or aggregation layer, so identical consecutive events can still flood the downstream receiver.
Webhook subscriptions support a rate‑limit header, but they do not collapse repeated identical events, meaning bursty failures can still produce multiple alerts even when the rate limit is respected.
Given these constraints, what mechanisms could be added to the Nomad event stream to deduplicate or aggregate similar events before they reach external alerting systems? How might the existing rate‑limit header be leveraged or extended to reduce repeated identical events? What trade‑offs arise when introducing server‑side aggregation versus client‑side filtering?