Journald watcher for instant pages or a periodic digest for runtime service failures on openSUSE?

Question

I run a small openSUSE Leap server and want to be notified only about actionable failures, not routine log traffic. Boot-time health is already covered by health-checker with snapper rollback on my btrfs root, which gives me a single meaningful signal when a boot goes bad. The gap is runtime failures, such as a systemd unit failing mid-day while the system keeps running.

Two documented approaches seem possible. One is a journald-based watcher (a systemd timer or persistent unit running journalctl with priority and unit filters like -p err and -u) that notifies as soon as a matching error appears. The other is batching the same filtered journal output into a periodic digest, so failures surface on a schedule instead of immediately.

My constraints: I am the only admin, so alert fatigue matters, but some services are customer-facing and a multi-hour digest delay could hurt. I also know journald rate limiting can drop bursts, which might affect either approach.

Which approach fits a single-admin setup where only a few units are truly urgent? Is there a documented way to page immediately for specific units while digesting everything else, without running two separate alerting pipelines?

Tasadduq Burney · Answer

Direct answerFor a single-admin openSUSE Leap server with a few customer-facing units, use systemd's built-in OnFailure= for immediate paging of critical units and a single timer-driven script for mixed-granularity digest of the rest. This avoids two separate alerting pipelines while keeping alert fatigue low.Confirmed facts vs likely explanationConfirmed: systemd supports OnFailure= on a unit which triggers a oneshot service when the unit enters the failed state. That hooks into the unit state machine and fires within seconds of a hard failure, independent of journalctl polling.Likely: journalctl polling with --since and -p err -u can provide a digest, but time parsing and rate limiting behavior can vary by systemd version on Leap. Journald rate limiting can drop bursts before a poller sees them, making polling less reliable for immediate detection than OnFailure=.Steps needed for this caseIdentify critical units. Add OnFailure for each in /etc/systemd/system or a drop-in: [Unit]
OnFailure=notify-critical@%i.serviceCreate a generic notification service /etc/systemd/system/notify-critical@.service: [Unit]
Description=Notify on %i failure
[Service]
Type=oneshot
ExecStart=/usr/local/bin/alert-critical %iCreate a hybrid script /usr/local/bin/journald-alert-hybrid that checks critical units with a short window and non-critical with a longer window, then sends one mail or webhook per run.Create a timer /etc/systemd/system/journald-alert.timer with OnCalendar for e.g. every 15 minutes and Persistent=true, pointing to the hybrid script via a service.Reload and enable: systemctl daemon-reload
systemctl enable --now journald-alert.timerKey considerationsOnFailure triggers only on failed state, not on restarts or transient errors.Test journalctl --since syntax on your Leap version before deploying.Rate limiting may affect polling; consider journald.conf settings for critical units only after testing.Missing diagnostic detail that changes recommendation: what notification transport is available on this host, e.g. local mail, systemd-cat, or a webhook. The script implementation depends on that.

Journald watcher for instant pages or a periodic digest for runtime service failures on openSUSE?

1 answer

Direct answer

Confirmed facts vs likely explanation

Steps needed for this case

Key considerations

Your perspective belongs here.

0 question comments