Azure Pipelines Cache@2 task interaction with self‑hosted agents and stale cache handling

Question

The goal is to guarantee that any change to a file that influences the build—such as a version file stored outside the cached directories—forces the Cache@2 task to treat the cache as missed, thereby preventing the reuse of stale data that could produce non‑deterministic builds.

On Microsoft‑hosted agents the cache is automatically removed after seven days of inactivity, while self‑hosted agents retain caches until the folder is manually deleted or the agent is re‑imaged. Because the Cache@2 task derives its key solely from user‑supplied inputs and does not perform content‑based hashing, a modification to an uncached file will not generate a new key unless the pipeline author updates the key or restore key manually, leaving the behavior of stale cache reuse unresolved.

What strategies can be employed to automatically incorporate version‑file changes into the cache key without altering the pipeline YAML each time?
How can teams enforce cache invalidation on self‑hosted agents when uncached inputs change, given the lack of built‑in content hashing?
Is there a recommended cleanup policy or automated process that balances cache reuse benefits with the risk of stale entries on long‑running self‑hosted agents?

Tasadduq Burney · Answer

Answer the Question First
To make a Cache@2 task react to changes in a file that is not part of the cache (e.g., a VERSION file), you can generate a hash of that file in a pre‑cache script, expose the hash as a pipeline variable, and reference that variable in the task’s key or restoreKey. The variable will change automatically whenever the file changes, forcing a cache miss and a fresh restore without editing the YAML each time.
Key‑generation strategy

Run a short script before the Cache@2 step:
# Bash example – works on Linux, macOS, and Git Bash on Windows
VERSION_HASH=$(sha256sum $(Build.SourcesDirectory)/VERSION | cut -d ' ' -f 1)
echo "##vso[task.setvariable variable=versionHash]$VERSION_HASH"

Reference the variable in the cache task:
steps:
- task: Cache@2
  inputs:
    key: 'npm|$(Agent.OS)|$(versionHash)'
    path: $(Pipeline.Workspace)/npm

When the VERSION file changes, $(versionHash) changes, the key changes, and the task reports a Cache miss.

Alternative: Stamp file
A lightweight trick is to touch a small “stamp” file whenever the version file changes. Include the stamp file’s path in the key:
# In a pre‑cache script
STAMP_PATH=$(Build.SourcesDirectory)/.cache_stamp
if ! cmp -s $(Build.SourcesDirectory)/VERSION $STAMP_PATH; then
  cp $(Build.SourcesDirectory)/VERSION $STAMP_PATH
fi

key: 'npm|$(Agent.OS)|$(Build.SourcesDirectory)/.cache_stamp'

Because the stamp file is part of the repository, the hash of its path will change only when the version file changes, again forcing a miss.
Enforcing invalidation on self‑hosted agents

Self‑hosted agents do not auto‑expire caches. Add a scheduled or post‑build job that deletes cache directories older than a configured threshold.

Use the Azure DevOps REST API to list cache entries and delete those whose lastAccessed timestamp exceeds your policy.

Alternatively, run a simple script on the agent that removes the cache folder if the current hash differs from a stored hash on disk.

Recommended cleanup policy
Balance hit‑rate and freshness with a two‑tier approach:

Time‑based retention on the server side: Set Retention days to 30 for self‑hosted caches via the Azure DevOps UI. This removes stale entries automatically.

Content‑based invalidation: Use the hash‑generation strategy above so that any relevant file change triggers a fresh cache immediately.

Keep a log of cache hits/misses in the pipeline logs to monitor the impact.

What’s still missing?
To fine‑tune the cleanup script, could you share which operating system your self‑hosted agents run on? This determines the exact path to the cache directory (e.g., _work/_temp on Windows vs /var/opt/azp/_work/_temp on Linux).

Azure Pipelines Cache@2 task interaction with self‑hosted agents and stale cache handling

1 answer

Answer the Question First

Key‑generation strategy

Alternative: Stamp file

Enforcing invalidation on self‑hosted agents

Recommended cleanup policy

What’s still missing?

Your perspective belongs here.

0 question comments