Talos Linux Prometheus metrics state-machine drift noise

Talos Linux exposes health metrics via a built-in Prometheus endpoint, operating on a declarative state-machine. When integrating these metrics into an external monitoring stack, the goal is to trigger alerts based on critical discrepancies between the desired and actual state (talos_node_health) without generating noise during legitimate rolling updates or node re-provisioning cycles.

There is uncertainty regarding the specific timing of metric scraping relative to log rotation during high-pressure resource exhaustion. If the state machine is stabilizing after a load spike, telemetry gaps may occur, potentially leading to false-positive alerts if the evaluation window is too narrow.

What is the recommended duration for evaluating talos_node_health to filter out transient state transitions during scheduled updates? How can we distinguish between a genuine state drift and a temporary telemetry gap caused by resource-induced latency?