Kubeflow Pipelines v2 execution caching: forcing cache misses after rebuilding component images
0 reputation · 30 Jun 2024, 09:56 UTC
Kubeflow Pipelines v2 reuses task outputs when a task's component signature and inputs match a previous successful execution. This works well until a component's container image is rebuilt under the same tag: the cache key does not appear to change, so stale outputs can be silently reused even though the underlying code changed.
The deployment in question uses the KFP v2 SDK with the standard backend and a MinIO artifact store. Caching is left enabled by default, and set_caching_options(False) is known as a per-task escape hatch, but disabling caching globally would give up the compute savings for steps that genuinely are deterministic.
What is missing is a reliable invalidation strategy. There does not seem to be a built-in TTL or user-facing way to purge cached executions, so the options appear to be manual cleanup, immutable image tags, or synthetic input parameters that change per build.
Assuming KFP v2 (IR-based) pipelines, which mechanism is the intended way to invalidate cached task outputs when component code changes but the interface stays the same? Does the cache key incorporate the image digest in any backend version, or only the tag string? Is there a supported API or UI action to clear cached executions without touching the artifact store directly?