Kubeflow Pipelines restore: verifying artifact integrity and ownership after metadata recovery
0 reputation · 02 Nov 2023, 11:15 UTC
We run a standard Kubeflow Manifests deployment where pipeline run metadata lives in the MySQL metadata database and run artifacts live in an external object storage bucket. We are designing a restore procedure that covers both, since a control-plane backup alone does not protect the artifacts.
The unresolved decision is verification. After restoring the metadata database and persistent volumes, Kubeflow appears to offer no native way to confirm that restored runs still reference existing artifacts, or that the artifact contents match what existed at backup time. There is also no reconciliation report comparing run IDs, parameters, and artifact URIs between the source and restored environment.
A second concern is the permission boundary: restored pipeline resources inherit the target cluster's RBAC and Kubeflow IAM bindings, which may not match the source environment, so original ownership and namespace isolation are not guaranteed.
Assuming Kubeflow Pipelines 2.x with external object storage:
- What is a practical way to validate that every restored run's artifact URIs resolve and match expected contents, given the lack of built-in checksums?
- How should original user ownership be preserved or audited when RBAC bindings differ between source and target clusters?
- Is re-running a sample pipeline from restored definitions a sufficient functional check, or are there metadata-level checks we are missing?