Prometheus–Kubernetes target lifecycle and time series retention after pod crash

Prometheus integrates with Kubernetes service discovery to scrape pod endpoints and logs scrape errors at warn level with target address and error context. The integration relies on metadata labels such as __meta_kubernetes_pod_name to associate failures with pod lifecycle events. Target state moves from UP to DOWN after consecutive scrape failures within scrape_interval, and the target is removed from active targets in the next cycle.

The precise handling of time series associated with a DOWN target is uncertain. Current behavior retains samples until explicit removal after the failure threshold, but the optimal policy for immediate purge versus grace period retention for post-mortem analysis remains under discussion. Version differences in failure thresholds and experimental feature flags can alter handling. relabel_configs and honor_labels may change target identity after failure.

What retention window applies to samples from a DOWN target before they become unqueryable? Does address rewriting via relabeling create duplicate series for the same logical pod? How should operators interpret persistent discovery labels after pod termination?