Kubeflow Pipelines and Prometheus metric exposure for selective failure alerting

The short answer

KFP does not label run failures as "user error" vs "platform error" — a run in Failed state almost always means the user's container exited non-zero, so alerting on run failure rate is the wrong signal for platform health. Instead, alert on component-level metrics (apiserver 5xx responses, ML Metadata gRPC connection errors, persistence agent sync failures, storage and MySQL errors, Argo controller reconciliation errors) for platform degradation, and treat user run failures only as an aggregate trend. For stuck runs, alert on Pending duration against a short scheduling SLA, and on Running duration against a per-pipeline historical percentile — never a fixed global limit.

What is confirmed vs. inferred

Confirmed behavior: KFP's API server and persistence agent expose run/workflow status counts labeled by state (Succeeded, Failed, Error), and the Argo workflow controller that actually executes pipelines exposes controller metrics such as work-queue depth, reconciliation errors, and workflow duration histograms. What is not provided by KFP is any failure-cause classification. The platform-vs-user distinction must be inferred by correlating workflow failures with component health: if Failed runs spike while apiserver 5xx, MLMD errors, and storage errors are flat, it is user code; if failures coincide with component error spikes, it is platform degradation.

Exact metric names and labels vary across KFP versions and deployment modes (standalone KFP vs. full Kubeflow), so treat every name below as a pattern to verify, not a guarantee.

Alerting on platform degradation

Build alerts on infrastructure signals, for example:

• HTTP 5xx rate on the KFP apiserver request metrics
• ML Metadata gRPC client/server error counters
• Persistence agent workflow sync failure counters
• MinIO/S3 operation errors and MySQL connection errors
• Argo controller reconciliation error rate and queue depth growth

For user failures, use a trend alert only — e.g., the Failed-state run rate doubling versus a trailing baseline over several hours — and route it to a dashboard or low-severity channel, not paging.

Detecting stuck runs without paging on long training jobs

Use two different thresholds for the two states:

• Pending: alert when a run stays Pending longer than your pod-scheduling SLA (typically a few minutes, padded for cluster autoscaler spin-up). Pending beyond that almost always means scheduling or quota problems, not user behavior.
• Running: compare current duration against a per-pipeline baseline, e.g. histogram_quantile over historical durations for that pipeline, and alert when a run exceeds something like the p99 of its own history. A fixed global limit will page you on every legitimate long training job.

Precompute per-state counts and per-pipeline duration baselines with recording rules so alert expressions stay cheap:

# illustrative only — verify metric/label names against your /metrics endpoints
- record: kfp:runs_by_state:sum
  expr: sum by (state) (run_status_count)

# stuck pending: pending longer than scheduling SLA
- alert: KFPRunStuckPending
  expr: (time() - max by (run_id) (run_start_pending_timestamp)) > 600
    and on(run_id) kfp:runs_by_state:sum{state="Pending"} > 0
  for: 5m

Verification before you trust the rules

1. Scrape or port-forward the apiserver, persistence agent, MLMD, and Argo controller /metrics endpoints and list the actual metric names and state labels in your deployment.
2. Fail a pipeline deliberately with a user-code error, then separately simulate platform failure (e.g., scale down MLMD or MySQL) and confirm which metrics move in each case.
3. Create a run with an unschedulable resource request to force a long Pending state and confirm the stuck-pending alert fires while a normal long training run does not.
4. Evaluate the alert expressions in Prometheus's expression browser over several days of history to estimate false-positive rates before routing to paging.

One caveat worth flagging: percentile-based Running alerts need enough historical runs per pipeline to be meaningful — newly created pipelines will produce noisy baselines, so consider suppressing duration alerts until a pipeline has a minimum run count.