Question
timeout waiting for pod to be ready when restoring Kubeflow Pipelines workflow-controller pod
Rahul Moss
0 reputation · 16 Oct 2023, 19:40 UTC
47.7K views1
Loading…
0 reputation · 16 Oct 2023, 19:40 UTC
The goal is to determine whether Velero‑based filesystem backups or CSI‑driven volume snapshots provide a reliable, application‑consistent restore path for Kubeflow Pipelines artifact stores (MLflow and MinIO) without triggering the 'timeout waiting for pod to be ready' error during workflow‑controller pod recovery.
The uncertainty stems from Velero restore failures that occur when the restored pod depends on a PVC that is not yet bound, because Velero does not wait for the underlying CSI snapshot to reach ReadyToUse before creating the pod. While CSI snapshots can reduce restore latency, they require identical storage class names between source and target clusters and careful handling of the KFP metadata database exclusion guidance, which varies across Kubeflow versions.
Which backup approach—Velero filesystem backup or CSI volume snapshot—ensures application‑consistent restores of KFP artifact stores without causing the workflow‑controller timeout? What storage class alignment steps are necessary when using CSI snapshots for the workflow‑controller PVC? How can post‑restore validation confirm that artifact links in MinIO and MLflow remain intact and accessible in the KFP UI?
31775 reputation · 17 Oct 2023, 01:32 UTC
To avoid the timeout waiting for pod to be ready error during Kubeflow Pipelines (KFP) restoration, CSI volume snapshots are the preferred approach over Velero filesystem backups (Restic/Kopia). While filesystem backups are more flexible, they often introduce significant latency during the data restoration phase, causing the workflow-controller to fail its readiness probes before the volume is fully populated and accessible.
The timeout typically occurs because Velero may create the Pod resource before the underlying CSI snapshot has transitioned to a ReadyToUse state. The workflow-controller attempts to initialize, fails to find its expected state on the PVC, and crashes or hangs, leading the restore process to time out while waiting for the Pod to signal readiness via its health check.
When using CSI snapshots to ensure the workflow-controller and artifact stores (MinIO/MLflow) recover correctly, follow these alignment steps:
kubeflowstorageClassName in the target cluster exactly matches the source cluster. If they differ, you must use a Velero ConfigMap for StorageClass mapping to prevent the PVC from remaining in a Pending state.VolumeSnapshotClass is defined in the target cluster and is bound to the same CSI driver used in the source.mysqldump) to avoid data corruption or state mismatch.To confirm that artifact links between MinIO, MLflow, and the KFP UI are intact, perform these checks:
kubectl get pods -n kubeflow.workflow-controller pod.To refine this recommendation: Are you restoring to a cluster with the same cloud provider and region as the source? If the region differs, CSI snapshots may require additional cross-region replication steps before Velero can bind them.
Use comments to ask for clarification. Post a solution as an answer.
31,775 reputation · 17 Oct 2023, 03:23 UTC
One clarification worth adding: the workflow-controller deployment itself is essentially stateless — it doesn't keep artifacts on its own PVC. So when it hits timeout waiting for pod to be ready during a Velero restore, the culprit is usually not its volume at all, but a missing dependency: the ml-pipeline API or MinIO service it connects to at startup, or a webhook/certificate that Velero restored out of order. Check kubectl describe pod and the container logs before assuming a PVC binding issue — FailedMount events point to storage, while repeated crash loops with connection errors point to service ordering.
If the PVC race is confirmed, a practical mitigation is restoring in two passes (volumes first, then workloads) so snapshots reach ReadyToUse before pods are scheduled. Behavior varies by Velero and KFP version, so verify against your release notes.