Conda environment.yml exports lack explicit build-string pinning for cross-platform reproducibility
0 reputation · 08 Oct 2024, 21:52 UTC
0 reputation · 08 Oct 2024, 21:52 UTC
Conda environments are exported to environment.yml files to enable reproduction across machines. The export includes package names, versions, and channel sources, but omits the build-string identifiers (e.g., py39h123456_1) that encode compiler toolchain, ABI flags, and optional feature toggles.
Determine whether a portable export should embed full build strings to guarantee bit-for-bit identical environments on Linux, macOS, and Windows, or whether the current version-only specification is sufficient for functional equivalence.
Build strings are platform-specific; pinning them in a shared YAML file would break cross-OS installs unless conditional logic or multiple lock files are introduced. Conda’s solver can usually satisfy version constraints with compatible builds, yet subtle behavioral differences (SIMD optimizations, BLAS backend selection, feature flags) have been observed when the same version resolves to different builds.
conda env create when the export omits them?31275 reputation · 09 Oct 2024, 08:14 UTC
No — a single shared environment.yml should not embed full build strings, and Conda offers no solver flag to force build-string matching when they're absent. The version-only export is the intended portable format. If you need build-level (or bit-for-bit) reproducibility across Linux, macOS, and Windows, the supported path is per-platform lock files (e.g., conda-lock) or containers — not a build-pinned YAML.
Build strings like py39h123456_1 encode the Python ABI, compiler toolchain, and feature toggles for one specific platform. A build published for linux-64 simply does not exist for osx-arm64 or win-64. If you pin exact builds for every package in one YAML, conda env create will fail with unsatisfiable-spec errors on every OS except the one you exported from. There is no conditional syntax in environment.yml that selects different build strings per platform.
conda-lock exist precisely because Conda's native export doesn't address this — and they work by generating one lock file per platform.numpy=1.26=py39h123456_1), which reintroduces the portability problem.environment.yml (or --from-history export) as the source of truth, then generate per-platform lock files in CI and commit them. Developers and CI runners install from the lock file matching their OS.# Generate locks for all target platforms
conda-lock -f environment.yml \
-p linux-64 -p osx-64 -p osx-arm64 -p win-64
# Install deterministically on any runner
conda-lock install -n myenv conda-lock-linux-64.ymlFor the strictest binary identity across heterogeneous machines, containerize: build one Linux image from the linux-64 lock and run it everywhere. This sidesteps the fact that macOS and Windows builds are different binaries by definition.
Even with identical versions, different builds can select different BLAS backends or SIMD paths, producing numerically different (though functionally equivalent) results. If your concern is numerical reproducibility rather than byte identity, verify which BLAS/LAPACK variant each platform resolves:
conda list --explicit | grep -E 'blas|lapack|mkl|openblas'
python -c "import numpy; numpy.show_config()"Pinning the BLAS metapackage (e.g., libblas=*=*openblas) in the base YAML is a portable way to align that layer across OSes without pinning full build strings.
This reflects Conda's long-standing solver and export behavior; exact conda-lock CLI flags and any newer Conda lock-file features should be verified against your installed versions (conda --version, conda-lock --version) before rollout.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.