Kubeflow Pipelines restore: verifying artifact integrity and ownership after metadata recovery

Question

We run a standard Kubeflow Manifests deployment where pipeline run metadata lives in the MySQL metadata database and run artifacts live in an external object storage bucket. We are designing a restore procedure that covers both, since a control-plane backup alone does not protect the artifacts.

The unresolved decision is verification. After restoring the metadata database and persistent volumes, Kubeflow appears to offer no native way to confirm that restored runs still reference existing artifacts, or that the artifact contents match what existed at backup time. There is also no reconciliation report comparing run IDs, parameters, and artifact URIs between the source and restored environment.

A second concern is the permission boundary: restored pipeline resources inherit the target cluster's RBAC and Kubeflow IAM bindings, which may not match the source environment, so original ownership and namespace isolation are not guaranteed.

Assuming Kubeflow Pipelines 2.x with external object storage:

What is a practical way to validate that every restored run's artifact URIs resolve and match expected contents, given the lack of built-in checksums?
How should original user ownership be preserved or audited when RBAC bindings differ between source and target clusters?
Is re-running a sample pipeline from restored definitions a sufficient functional check, or are there metadata-level checks we are missing?

Tasadduq Burney · Answer

Validating Artifacts and Ownership After Metadata Recovery
Kubeflow Pipelines (KFP) 2.x decouples metadata (stored in MySQL/MLMD) from actual artifact data (stored in object storage). Because the metadata database stores URIs but not content checksums, the platform cannot natively verify if a restored URI still points to the correct, unaltered file.

Artifact Integrity Verification
To validate that restored runs reference existing and correct artifacts, you must implement an external reconciliation script. Since KFP lacks built-in checksums, the only way to ensure content integrity is to compare the restored state against a manifest generated at the time of backup.

Extract URIs: Query the restored MLMD database (joining artifacts and executions tables) or use the KFP API list_runs to enumerate all artifact URIs for the restored period.
Existence Check: Perform a HEAD request for each URI against the object store to confirm the object exists.
Content Validation: Compare the object size and ETag/MD5 against your backup-time manifest. Note: ETags may not be simple MD5s for multipart uploads; treat them as heuristics unless your upload method is consistent.

Preserving and Auditing Ownership
Restored pipeline resources inherit the RBAC and IAM bindings of the target cluster. If the target cluster has different Profiles or RoleBindings, original ownership is not automatically preserved.

Audit Mapping: Export Profiles, RoleBindings, and AuthorizationPolicies from the source cluster.
Diff Analysis: Compare these exports against the target cluster's current state to identify users who have lost access to their restored namespaces or runs.
Remediation: Manually apply missing RoleBindings to the target cluster to align ownership with the source environment.

Functional vs. Metadata Verification
Re-running a sample pipeline is a necessary smoke test for the control plane and write-paths, but it is insufficient for a full restore validation. It proves the system can create new data, but not that historical data is intact.
A complete verification requires combining the functional test with metadata-level checks: comparing row counts across MLMD tables (executions, artifacts, events) between the source dump and the restored database to ensure no data loss occurred during the import.

Verification Commands
To verify the connectivity of a specific restored artifact, use a scoped cloud CLI command (example for AWS S3):
# Verify existence and size of a restored artifact
aws s3api head-object --bucket [your-bucket] --key [artifact-path-from-db]

Missing Diagnostic Detail: Do you have a pre-existing manifest of artifact hashes/sizes from the time of backup? If not, verification is limited to existence checks (URI resolution) rather than integrity checks (content matching).

Kubeflow Pipelines restore: verifying artifact integrity and ownership after metadata recovery

1 answer

Validating Artifacts and Ownership After Metadata Recovery

Artifact Integrity Verification

Preserving and Auditing Ownership

Functional vs. Metadata Verification

Verification Commands

Your perspective belongs here.

0 question comments