Cache Granularity vs Build Speed: Per‑Dependency vs Full Requirements Cache Invalidation
0 reputation · 27 Mar 2024, 06:12 UTC
0 reputation · 27 Mar 2024, 06:12 UTC
Determine the optimal cache invalidation strategy for Read the Docs builds that balances build speed against the risk of serving stale documentation when dependencies change.
Read the Docs uses an internal hash of the requirements.txt to decide whether to reuse cached build artifacts. The hashing algorithm is not exposed, and there is no public API to trigger partial cache purges. It is unclear whether the system can detect a single dependency update or if any modification to the file forces a full cache rebuild.
requirements.txt changes, or can it detect and react to specific dependency updates?34775 reputation · 27 Mar 2024, 16:01 UTC
Read the Docs historically invalidates the entire build environment cache whenever requirements.txt changes. Any edit—even a single version bump—forces a full reinstall of the Python environment. There is no documented per-dependency cache granularity, no configuration setting to enable it, and no public API to trigger a partial purge.
Established behavior: The platform appears to use a full-file hash as the cache key. Because the algorithm is internal and undocumented, you cannot predict invalidation from the file contents alone.
Uncertain: It is unclear whether recent platform updates have introduced finer-grained invalidation without public documentation. Do not assume per-dependency behavior unless you observe it directly in build logs.
requirements.txt.If your build logs show that only the changed package is reinstalled after a single-line edit, share that detail. It would shift the recommendation from “expect full rebuilds” to “verify that granular caching is active and reliable.”
Since granularity is not configurable, the practical lever is dependency-tree size. A smaller requirements.txt reduces the cost of each full rebuild. If your plan supports custom build images, pre-installing dependencies in the image can avoid the requirements.txt cache path, but that moves update responsibility into your image pipeline.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 27 Mar 2024, 08:05 UTC
Internally the platform hashes the entire requirements.txt file. Because the hash is a function of the whole text, any line change— even a single version bump—forces a full rebuild of the Python environment. The build worker is stateless, so the pip wheel cache is not shared across runs; the installation step always pulls wheels again, regardless of whether the dependency itself changed.
So, while Read the Docs does not expose per‑dependency granularity, careful dependency management still reduces rebuild time.