Julia package manager and JIT compiler precompilation cache reuse at integration boundary
0 reputation · 21 Sept 2022, 22:55 UTC
0 reputation · 21 Sept 2022, 22:55 UTC
At the integration boundary between Julia's package manager and the JIT compiler, precompiled code is stored in a per-environment depot cache to accelerate subsequent using times. Cache entries are keyed by package version, manifest state and Julia version, and invalidation is driven mainly by version and manifest changes.
The design goal is to understand when cached artifacts are reused versus rebuilt and the resulting stale result risk for downstream code. Source edits that do not change version or manifest can leave previously compiled artifacts in use, and there is no fully automatic guarantee that a change in a transitive dependency forces immediate invalidation of all dependents.
REPL integration with live code update tools operates outside the precompilation cache, creating a mixed execution state where some call sites may use cached images and others use revised source. Behavior is version sensitive across Julia releases and changes in package manager semantics, and custom sysimages or PackageCompiler images alter caching assumptions.
What conditions under default depot layout and precompilation settings cause the package manager to invalidate a precompile cache entry when a transitive dependency changes? How does interaction with live code update tools affect which call sites use cached images versus revised source? What version assumptions govern cache reuse after source edits without a manifest change?
35275 reputation · 22 Sept 2022, 07:04 UTC
Under the default depot layout, the package manager invalidates a precompile cache entry for a package when that package’s own version changes or when its manifest entry changes. A change in a transitive dependency does not, by default, invalidate the dependent’s cache entry if the direct dependency’s version and manifest entry remain unchanged. The cache key combines package version, manifest state, and the running Julia version; transitive dependency identity is not a standalone invalidation trigger under default settings.
Confirmed: cache entries are stored per environment in the depot and are keyed by package version, manifest state, and Julia version. Confirmed: source edits that do not alter version or manifest can leave previously compiled artifacts in use. Likely explanation: when a transitive dependency is updated but the direct dependency’s manifest entry is untouched, Pkg has no recorded reason to rebuild the dependent’s precompiled image, so the stale image may persist until a manifest or version change forces a rebuild.
Tools such as Revise.jl operate outside the precompilation cache. They track source files by path and modification time and redefine bindings in the running session. Call sites that already resolved to functions from a cached precompiled image continue using that image unless the tool explicitly redefines those bindings. New invocations after an edit may execute revised source, while older call sites still reference cached compiled code. This produces a mixed execution state where some call sites use cached images and others use revised source. Revise does not automatically rebuild the precompile cache.
Cache reuse assumes the package version, manifest entry, and Julia version remain unchanged. Source edits without a manifest change can still produce stale results if exported function signatures change, because callers compiled against the old signatures may continue linking to the old compiled code. Internal-only changes carry lower but not zero stale-result risk. Custom sysimages or PackageCompiler images alter these assumptions and are outside default depot behavior.
julia -e 'using Pkg; println(DEPOT_PATH)'Pkg.precompile(). Deleting the specific compiled cache file is a scoped alternative.Are you observing stale behavior after editing a transitive dependency’s source while its version and manifest entry remain unchanged? This determines whether the fix is a manifest bump or cache deletion, or simply expected live-update mixed-state behavior.
Behavior is version-sensitive across Julia releases and can differ with custom depot paths, sysimages, or PackageCompiler builds. The exact internal heuristic for transitive dependency invalidation is not treated as a documented contract here. This draft requires current verification against your Julia version before operational use.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.