Real‑time health_status event tailing versus periodic inspect polling for low‑notification‑noise health alerts

Direct answer

No documented, version-stable measurable latency exists between a healthcheck outcome and its visibility in podman events versus podman inspect for rootless containers; do not quote a millisecond SLA without measuring it in your environment. Intermittent event-stream loss makes health_status tailing unreliable by itself because disconnections and reconnects can miss or delay notifications. The low-noise pattern is to tail events for immediate transitions, run one podman inspect reconcile on every connect/reconnect and at a long fixed interval, and alert only when the inspected health state differs from the last alerted state.

Confirmed behavior and uncertainty

Confirmed: podman inspect is point-in-time and reports the last health state stored by Podman/conmon. podman events health_status is a change notification, not guaranteed delivery. Stream disconnections, reconnects, and buffering can miss or delay notifications, particularly across rootless session restarts. Healthcheck interval, timeout, and retries typically dominate the time between a failing check and a visible unhealthy state.

Uncertain / unverified: No published, version-stable measurable latency exists for a healthcheck outcome to appear in podman events versus podman inspect for rootless containers. Any specific millisecond figure is unverified. Rootless scheduling may add jitter, but no documented rootless-specific lag difference between the two paths is established.

Steps for this case

1. Confirm the container has a healthcheck and that this Podman build emits health_status events.
2. Tail events for immediate transition alerts:

podman events --filter event=health_status --format '{{.Time}} {{.Status}} {{.ID}} {{.From}}'

1. On connect or reconnect, run one reconcile inspect and compare against the last alerted state:

podman inspect --format '{{.State.Health.Status}} {{.State.Health.FailingStreak}}' <container-id>

1. Add a long-interval inspect poll, for example every 5 to 10 minutes, only as a safety net after stream loss.
2. Emit an alert only when State.Health.Status changes from the previously alerted value; suppress repeated alerts for the same unhealthy state.

Verification

Run a rootless container with a healthcheck that writes completion time to a bind-mounted file. Capture podman events, grep for health_status, and poll podman inspect at a fixed interval. Compare the healthcheck completion timestamp, event timestamp, and inspect poll timestamp across repeated failures and recoveries. Repeat with the event stream stopped during a failure; after reconnect, the single reconcile inspect should recover the missed transition without duplicate alerts.

Missing diagnostic detail

What Podman version and build are you running, and have you confirmed that health_status events are emitted in this rootless build? If they are not emitted, drop event tailing and use transition-detecting inspect polling instead.