503 Service Unavailable in Kubeflow after upgrade: Istio routing misconfiguration

Answer

To eliminate the 503 responses, restore the Istio objects that were generated by the known‑good Kubeflow overlay and verify that the CRDs remain compatible with that version before re‑applying them.

Likely explanation

The upgrade regenerated Istio VirtualService, DestinationRule and Gateway manifests, but the existing CRDs were not fully reconciled, leaving routes that the gateway cannot match.

Confirmed facts

• Istio 503 errors usually indicate missing or mismatched route configuration at the ingress gateway.
• Kubeflow upgrades that use Kustomize overlays can be rolled back by re‑applying the previous overlay.
• PVC‑based workloads (MySQL, Minio) retain data as long as the volume claims are not deleted.

Steps to revert the Istio configuration

1. Backup current Istio resources:

   kubectl get virtualservice -n istio-system -o yaml > istio-vs-backup.yaml
   kubectl get destinationrule -n istio-system -o yaml > istio-dr-backup.yaml
   kubectl get gateway -n istio-system -o yaml > istio-gw-backup.yaml
   

2. Re‑apply the previous Kubeflow Kustomize overlay (the one that produced the working Istio manifests):

   cd /path/to/kubeflow/manifests
   kustomize build overlay-previous | kubectl apply -f -
   

3. If the overlay does not contain the Istio resources, restore the backed‑up YAMLs from step 1:

   kubectl apply -f istio-vs-backup.yaml
   kubectl apply -f istio-dr-backup.yaml
   kubectl apply -f istio-gw-backup.yaml
   

Verifying CRD compatibility before re‑applying

1. List the CRDs currently installed:

   kubectl get crd -o name | grep -E 'kubeflow|tfjob|pytorchjob' 

2. Fetch the CRD definitions shipped with the target Kubeflow version (e.g., from the release tarball or GitHub tag):

   curl -L https://github.com/kubeflow/manifests/releases/download/v1.4.0/kubeflow-crds.yaml -o target-crds.yaml
   

3. Compare schemas; ensure no fields have been removed or altered incompatibly. A quick diff can highlight missing fields:

   kubectl get crd tfjobs.kubeflow.org -o yaml > current-tfjob.yaml
   diff -u target-crds.yaml current-tfjob.yaml | grep '^-' || echo "No removed fields detected"
   

4. If the diff shows removed fields that are still present in existing objects, either edit the objects to drop those fields or upgrade the CRDs to a version that retains them before proceeding.

Backup strategy for MySQL and Minio metadata

• Snapshot the PVCs (assuming the storage class supports snapshots):

  # Example using Velero
  velero backup create kubeflow-pre-rollback --include-namespaces kubeflow --snapshot-volumes
  

• If Velero is unavailable, use the native cloud provider snapshot tool (e.g., AWS EBS snapshots, GCP PD snapshots) while keeping the PVCs bound to the pods.
• Verify the snapshot can be restored by creating a test PVC from the snapshot and checking that the data is accessible.
• During the rollback, do **not** delete the MySQL or Minio PVCs; leave them bound so the pods can re‑attach to the same volumes after the Istio objects are restored.

Missing diagnostic detail

To confirm that the previous overlay is available, do you have access to the Kustomize overlay that was used before the upgrade (e.g., in version control or a backup)? If not, the restoration path will need to retrieve that overlay from your source repository.