Julia environment reproducibility across disparate operating systems
0 reputation · 15 Jun 2022, 09:53 UTC
0 reputation · 15 Jun 2022, 09:53 UTC
Julia utilizes Project.toml for direct dependencies and Manifest.toml to lock the entire dependency tree, ensuring that Pkg.instantiate() can recreate a specific environment state.
A challenge arises when distributing these files across different operating systems. While the package versions are tracked via UUIDs, certain binary artifacts and platform-specific dependencies may cause inconsistencies when a manifest generated on one OS is instantiated on another.
When targeting a cross-platform development workflow (e.g., Linux and macOS), it is unclear how to maintain a single source of truth for the environment without introducing platform-specific resolution conflicts.
Manifest.toml files in a multi-OS team to prevent resolution errors during instantiation?No, Julia does not support a single platform-agnostic Manifest.toml for binary dependencies. The manifest records exact versions of jll packages (binary artifacts) that differ by OS and architecture. A manifest generated on Linux will fail or pull incompatible binaries when instantiated on macOS or Windows.
Recommended multi-OS strategy: Commit only Project.toml to version control. Let each developer and CI runner generate its own Manifest.toml locally via Pkg.instantiate(). For applications that require reproducible binary versions per platform, commit per-platform manifests using a naming convention (e.g., Manifest.toml.linux, Manifest.toml.macos) and restore the correct one in CI.
Manifest.toml locks the full dependency tree, including jll packages that wrap binary artifacts (OpenBLAS_jll, GR_jll, HDF5_jll, etc.).jll package publishes distinct builds per platform (Linux x86_64, macOS arm64, Windows x86_64, etc.) with different UUIDs, versions, and hashes.Pkg.instantiate() reads a manifest, it expects the recorded binary artifacts to exist for the current platform. Cross-platform mismatches cause resolution errors or silent selection of wrong binaries.Project.toml only.Manifest.toml.Pkg.instantiate() on their platform to resolve a valid manifest.Project.toml.Pkg.instantiate() on each target OS (Linux, macOS, Windows) to generate a platform-specific Manifest.toml.Manifest.toml.linux, Manifest.toml.macos, Manifest.toml.windows for auditability.Manifest.toml before instantiating.Pkg.instantiate() fails with "artifact not found" or checksum errors for any jll dependency.# On Linux
julia --project -e 'using Pkg; Pkg.add("Zlib_jll"); Pkg.instantiate()'
cp Manifest.toml Manifest.toml.linux
# On macOS (or in a macOS CI runner)
julia --project -e 'using Pkg; Pkg.instantiate()'
cp Manifest.toml Manifest.toml.macos
# Diff the two manifests
diff -u Manifest.toml.linux Manifest.toml.macos
You will see different UUID/version/hash entries for Zlib_jll and its dependencies.
Does your project have any direct or transitive dependency on a jll package (binary artifact)? Run Pkg.status() after instantiate on each OS and check for packages ending in _jll. If the answer is "no" (pure-Julia tree), a single committed Manifest.toml can be shared—but this is fragile because a future dependency update may introduce a binary artifact without notice.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.