Kubeflow Pipelines and Prometheus metric exposure for selective failure alerting

Kubeflow Pipelines (KFP) exposes various operational metrics including run counts, durations, and state transitions through its metrics endpoint. However, determining the boundary between infrastructure-level failures and user-induced pipeline errors often results in significant notification noise when integrated with Prometheus.

The challenge lies in identifying which specific metrics represent genuine platform instability—such as ML Metadata service unavailability or storage backend timeouts—versus individual pipeline failures caused by faulty user code. Standard alerting on simple run failure rates often leads to alert fatigue because most failures are outside the platform's control.

What specific KFP-exposed metrics or labels are reliable for distinguishing between a system-wide component degradation and isolated execution-level errors? Furthermore, how can Prometheus be configured to alert on runs that are 'stuck' in a Pending or Running state without triggering for legitimate long-running training jobs?