Pulsar per-topic vs aggregated backlog alerting trade-offs
0 reputation · 20 Dec 2021, 09:46 UTC
Context
Apache Pulsar 2.11+ exposes pulsar_broker_msg_backlog with topic and namespace labels, enabling granular alerting. The official documentation recommends alerting on backlog thresholds but provides no universal values and does not resolve whether operators should alert on per-topic series or pre-aggregated namespace/cluster totals.
Constraints
Per-topic rules generate high cardinality that can overwhelm Prometheus; recording rules can aggregate first but add latency and may mask tenant-level issues. Transient producer bursts and bundle splits cause short-lived spikes that trigger noise if evaluation windows are too short. Geo-replication environments need region-pair context because a single global threshold misfires on high-latency links. Metric names and availability shift across Pulsar 2.10, 2.11, and 3.0 releases, so any rule set must be version-pinned or use relabeling.
Goal
Define a practical alerting strategy that balances actionability for tenants against Prometheus cardinality limits and notification noise, while remaining maintainable across Pulsar version upgrades.
What cardinality ceiling should guide the decision to aggregate at namespace level versus retaining topic labels? Which multi-window evaluation patterns (e.g., "for: 15m") have proven effective at suppressing bundle-split and burst noise without delaying real incident detection? How are operators handling the absent compaction-lag guidance for pulsar_topic_compaction_backlog in 2.11+?