Kubeflow Pipelines v2 execution caching: forcing cache misses after rebuilding component images

Question

Kubeflow Pipelines v2 reuses task outputs when a task's component signature and inputs match a previous successful execution. This works well until a component's container image is rebuilt under the same tag: the cache key does not appear to change, so stale outputs can be silently reused even though the underlying code changed.

The deployment in question uses the KFP v2 SDK with the standard backend and a MinIO artifact store. Caching is left enabled by default, and set_caching_options(False) is known as a per-task escape hatch, but disabling caching globally would give up the compute savings for steps that genuinely are deterministic.

What is missing is a reliable invalidation strategy. There does not seem to be a built-in TTL or user-facing way to purge cached executions, so the options appear to be manual cleanup, immutable image tags, or synthetic input parameters that change per build.

Assuming KFP v2 (IR-based) pipelines, which mechanism is the intended way to invalidate cached task outputs when component code changes but the interface stays the same? Does the cache key incorporate the image digest in any backend version, or only the tag string? Is there a supported API or UI action to clear cached executions without touching the artifact store directly?

Tasadduq Burney · Answer

Answer
In Kubeflow Pipelines v2 the cache key for a task is a hash of the component specification, which includes the container image URI exactly as written in the component definition. The hash uses only the image tag string; it does not resolve the image digest. Therefore rebuilding and pushing a new image with the same tag leaves the cache key unchanged, causing the pipeline to reuse the previous task output.
Confirmed facts

The cache key does not incorporate the image digest in any released KFP v2 version (tested up to v2.2.0).
There is no built‑in TTL, automatic expiration, or user‑facing API/UI button to purge cached executions.
The only supported ways to change the cache key are to modify something that is part of the component spec – most commonly the image tag or an input parameter.

Recommended steps to force a cache miss

Use an immutable image tag – rebuild the component and push it with a tag that encodes the build identifier, e.g. a SHA digest (myrepo/mycomponent@sha256:abcd…) or a versioned tag like v1.2.3-. Update the component definition to reference this new tag and re‑submit the pipeline.
Add a dummy input parameter – introduce a parameter that does not affect the component logic (for example cache_buster) and give it a value that changes per build, such as a timestamp or a monotonically increasing counter. Because the parameter value is part of the cache key, the task will be re‑executed while caching remains enabled for all other steps.

Verification (optional)

# 1. Run pipeline with tag "latest"
# 2. Rebuild image, push with same tag "latest"
# 3. Re‑run pipeline – observe same artifact URLs and timestamps (cache hit)
# 4. Change tag to "latest-$(git rev-parse --short HEAD)" or add dummy param
# 5. Re‑run pipeline – new artifact URLs and later timestamps (cache miss)

If you cannot modify the component definition (e.g., it is consumed from a shared library), the immutable‑tag approach is the only viable option; otherwise the dummy parameter is simpler to implement.

Kubeflow Pipelines v2 execution caching: forcing cache misses after rebuilding component images

1 answer

Answer

Confirmed facts

Recommended steps to force a cache miss

Verification (optional)

Your perspective belongs here.

0 question comments