Pulsar per-topic vs aggregated backlog alerting trade-offs

Alerting Strategy: Aggregation vs. Granularity

The decision to aggregate at the namespace level or retain per-topic labels depends on your total topic count. As a general rule, if your environment exceeds 10,000 active topics, raw per-topic alerting on pulsar_broker_msg_backlog typically creates unsustainable cardinality for Prometheus, leading to increased memory pressure and slower alert evaluation cycles.

The Recommended Hybrid Approach

To balance tenant visibility with system stability, implement a tiered alerting strategy using Prometheus recording rules:

• Tier 1 (Namespace Aggregation): Create a recording rule that sums backlog by namespace. Alert on this aggregate to detect systemic consumer failures or tenant-wide saturation.
• Tier 2 (High-Value Topic Monitoring): Maintain per-topic alerts only for a curated list of "critical" topics (e.g., payment processing, auth logs) using a specific label or naming convention.
• Tier 3 (Exploratory Debugging): Retain raw per-topic metrics for short-term retention to allow operators to drill down into the specific "poison pill" topic once a namespace alert triggers.

Suppressing Noise from Bundle Splits and Bursts

Transient spikes caused by bundle splits or producer bursts can be suppressed using a multi-window evaluation pattern. A 15-minute for duration is generally effective for non-critical paths, as bundle reassignments typically resolve within minutes. For critical paths, use a dual-threshold approach:

# Critical: High threshold, short window
expr: pulsar_broker_msg_backlog > 1000000
for: 5m

# Warning: Lower threshold, longer window
expr: pulsar_broker_msg_backlog > 100000
for: 15m

Handling Compaction Backlog

Because pulsar_topic_compaction_backlog lacks universal guidance in Pulsar 2.11+, operators should avoid global thresholds. Compaction lag is business-logic dependent; a backlog in a configuration topic is negligible, while a backlog in a state-store topic can delay recovery. The practical approach is to define thresholds per-namespace based on the expected compaction frequency of the data.

Verification and Implementation

To determine if you have hit the cardinality ceiling, run the following query in your Prometheus expression browser to check the current series count for the backlog metric:

count(count by (topic) (pulsar_broker_msg_backlog))

Missing Diagnostic Detail: Are you utilizing a shared cluster with multiple tenants, or is this a single-tenant deployment? In multi-tenant environments, the recording rules must include the tenant label to prevent cross-tenant masking.