systemd journal and Linux kernel PSI integration: configuring useful alerts without notification noise

Configuring PSI Alerts for Signal over Noise

To balance sensitivity and noise when alerting on Pressure Stall Information (PSI), base your critical alerts on full and your early warnings on some. For most production workloads, an avg10 threshold of 10% to 20% is the recommended starting point to filter out transient spikes while capturing sustained resource contention.

Metric Selection: Some vs. Full

The choice depends on the desired alert severity:

• PSI some: Tracks time where at least one task is stalled. This is highly sensitive and ideal for early warning systems. However, in multi-threaded applications, it can be noisy as individual threads frequently block on I/O.
• PSI full: Tracks time where all non-idle tasks in the cgroup are stalled. This indicates a complete lack of progress and is the most reliable signal for critical resource exhaustion.

Aggregation and Thresholds

To avoid burst-induced false positives, use a two-tier smoothing approach:

1. Kernel Level: Use the avg10 metric. This provides a 10-second moving average directly from the kernel, which naturally smooths out micro-bursts.
2. Pipeline Level: Set your monitoring aggregation window (e.g., via Prometheus node_exporter or a custom script) to 1–5 minutes. An alert should only fire if the avg10 value exceeds your threshold consistently across multiple collection cycles.

Preventing Journald Rate-Limit Drops

If you are piping PSI events into the systemd journal, journald may silently drop logs during high-pressure events due to default rate limiting. To ensure visibility, adjust the following settings in /etc/systemd/journald.conf:

# Increase burst limit to prevent dropping pressure events
RateLimitIntervalSec=30s
RateLimitBurst=1000

After modifying the configuration, apply changes with systemctl restart systemd-journald.

Verification and Diagnostics

Verify that PSI is enabled on your kernel by checking for the existence of the pressure files:

ls /proc/pressure/memory /proc/pressure/cpu /proc/pressure/io

To validate your alert pipeline, use stress-ng to simulate sustained pressure and monitor the journal for suppressed messages:

journalctl -u your-monitoring-service | grep -i "suppressed"

Missing Diagnostic Detail: To further refine the avg10 threshold, please provide the specific workload type (e.g., database, JVM-based microservice, or high-throughput I/O) as memory pressure thresholds vary significantly between heap-heavy and disk-heavy applications.