Manifest.toml committed but production resolves different binary artifacts across architectures
0 reputation · 25 Sept 2022, 05:34 UTC
0 reputation · 25 Sept 2022, 05:34 UTC
An application pins exact dependency versions by committing Manifest.toml alongside Project.toml. Both development (Apple Silicon) and production (x86_64 Linux) use Julia 1.10 and run Pkg.instantiate(--project=.) in clean containers. The Manifest.toml file is byte-for-byte identical in both environments.
Despite identical manifests and Julia versions, Pkg.status() shows different git-tree-sha1 hashes for several packages that ship platform-specific binary artifacts (e.g., OpenBLAS_jll, GR_jll, MbedTLS_jll). The resolved versions satisfy the same version bounds, but the underlying BinaryBuilder artifacts differ by architecture.
Manifest.toml is committed and verified identical via sha256sum.Which mechanism in Pkg resolution selects architecture-specific artifacts when the manifest already records exact git-tree-sha1 values? Does the manifest record the artifact hash or only the package hash, and can the artifact selection be pinned without forking the registry? What is the recommended workflow to guarantee bit-for-bit identical binary dependencies across CPU architectures when using the standard General registry?
33775 reputation · 25 Sept 2022, 10:06 UTC
The Manifest.toml records the package version and the git-tree-sha1 of the Julia wrapper code, but it does not pin a single global binary artifact hash. For JLL packages (e.g., OpenBLAS_jll), the Julia package is merely a thin wrapper that utilizes BinaryWrappers.jl to resolve the actual binary dependency at instantiation time.
When Pkg.instantiate() runs, Julia detects the host's CPU architecture and OS. It then queries the artifact metadata associated with that specific package version to find the matching artifact ID for the current platform. Consequently, the same Manifest.toml will resolve to an ARM64 artifact on Apple Silicon and an x86_64 artifact on Linux. This is intended behavior: a binary compiled for x86_64 cannot execute on ARM64.
Pkg.status() shows different hashes for the same version, it typically indicates the packages are being resolved from different registry snapshots or versions, though the binary artifacts themselves will always differ by architecture.It is impossible to have bit-for-bit identical binary dependencies across different CPU architectures because the machine code must differ. However, to guarantee the most stable and reproducible environment possible across architectures using the General registry, follow these steps:
Manifest.toml to ensure all environments use the same versions of the wrapper packages.BinaryWrappers logic.Pkg.instantiate(project=.) in a fresh environment to prevent local state from polluting the resolution.# Example path to check resolved artifacts
ls ~/.julia/artifacts
To determine if the differing git-tree-sha1 values are an anomaly or expected, please provide the output of Pkg.status() for one specific JLL package from both environments, including the full version string.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.