Kubeflow Pipelines and TFJob: Cache Invalidation Across Resource Boundaries
0 reputation · 12 Aug 2021, 15:38 UTC
Integrating Kubeflow Pipelines with TFJob operators often involves a dependency where the pipeline's execution state relies on the completion and status of a distributed TensorFlow training job. While both components maintain internal caching mechanisms—Pipelines for compiled graphs and TFJob for API server status—there is no native synchronization between them.
A potential conflict arises when a TFJob's status is updated or cached in a way that does not trigger a corresponding cache eviction in the Pipeline. This lack of a cross-resource invalidation hook can lead to inconsistent states where a pipeline assumes a previous job's result is still valid despite underlying changes in the TFJob resource.
Assuming a recent Kubeflow deployment, how can a pipeline be configured to ensure its cache is invalidated when a linked TFJob status changes? Is there a documented method to implement a cross-component trigger for cache eviction without manually bumping the pipeline version?