503 Service Unavailable in Kubeflow after upgrade: Istio routing misconfiguration

503 Service Unavailable in Kubeflow after upgrade: Istio routing misconfiguration

During a recent upgrade of a Kubeflow deployment that uses Kustomize manifests, the cluster started returning 503 Service Unavailable responses from the Ingress gateway. The issue appears to be linked to Istio’s virtual service and destination rule definitions that were regenerated by the new overlay, but the existing CRDs were not fully reconciled.

Because Kubeflow does not ship a built‑in rollback, a failed upgrade requires manual restoration of the previous manifests and careful re‑application of stateful component configurations. The lack of a native CRD field removal process further complicates downgrades, as Kubernetes preserves the old schema.

Given these constraints, what steps can be taken to safely revert the Istio configuration without data loss? How can one verify that the CRD definitions are compatible with the previous Kubeflow version before re‑applying them? What backup strategy ensures that MySQL and Minio metadata remain intact during a rollback?