Reproducible Anaconda Environments: conda-lock vs environment.yml

The Problem: Silent Dependency Drift

A data science project shared through an environment.yml file usually states intent, not state. A line such as pandas>=2.0 tells the solver which versions are acceptable, but the conda solver resolves dependencies at installation time. Two developers can run the same command on different days or different operating systems and receive different package versions or build strings. A quiet update to a low-level binary such as libblas or mkl can then produce different numerical results between a developer's laptop and a CI runner.

The useful takeaway is to separate dependency intent from dependency resolution. For teams of more than two people, or any project that deploys to CI and production, a lockfile workflow is the safer default because it records exact versions, build strings, and hashes before anyone installs anything.

Decision and Constraints

Decision: adopt conda-lock to generate platform-specific lockfiles from a base environment.yml, then install from those lockfiles instead of resolving at create time.

Constraints driving this decision:

• Binary dependencies such as BLAS and MPI must remain identical across machines.
• CI jobs need predictable install times and offline-capable downloads.
• More than one developer maintains the project, so environment drift is a coordination risk.
• The project targets more than one operating system or architecture.

Supported Options Compared

Capability	environment.yml with conda	environment.yml with conda and pip	conda-lock lockfile
Resolution timing	At environment creation	At environment creation	Once at lockfile generation
Exact build strings	No	No	Yes
Package hashes	No	No	Yes
Cross-platform targets	One solve per platform	One solve per platform	Multiple platforms in one run
Git diff review	Easy, but only shows intent	Easy, but only shows intent	Detailed, shows every dependency change
Solver speed at install	Slow	Slow	Fast; solver is bypassed

Mamba can accelerate the solve for environment.yml, but it does not produce a lockfile. It improves speed without solving the drift problem.

Trade-offs

conda-lock adds a required generation step. Changing a dependency in environment.yml requires regenerating and committing the lockfile before teammates can install the update. The lockfile is also larger than a simple YAML manifest because it contains URLs and SHA256 hashes.

The benefit is reproducibility. Installing from a lockfile bypasses the solver, so channel priority changes or new package releases cannot silently alter the environment. It also supports offline installation when the packages are available in a local channel or mirror.

A lockfile is platform-specific. A linux-64 lockfile will not install on osx-arm64. You must generate a lockfile for every target platform your team or CI uses.

Implementation Workflow

1. Write the intent file

Create environment.yml in the project root. Keep most version constraints broad, but pin libraries with known breaking changes.

name: project-env
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - pandas
  - scikit-learn
  - pip:
    - xgboost==2.0.0

Run this step in your project directory with write access to the filesystem. No conda environment is created yet.

2. Pin the tool version

conda-lock CLI flags and output format change between releases. Pin the exact version in CI and document it for developers. Install it into your base conda environment or a dedicated tooling environment.

# Run in terminal; requires permission to install packages into conda
conda install -c conda-forge conda-lock=2.2.2

Replace 2.2.2 with the version your team has validated. Do not rely on latest in CI.

3. Generate lockfiles for each target platform

Run conda-lock against environment.yml. This solves the dependency graph once per platform and writes exact URLs and hashes.

# Run in terminal from the project root
# Requires write permission to the current directory
conda-lock -f environment.yml -p linux-64 -p osx-arm64

Expected result: A file named conda-lock.yml appears in the current directory. It contains resolved entries for linux-64 and osx-arm64, including SHA256 hashes and build strings. This is computationally expensive because it solves for every listed platform.

Risk: If a channel is unreachable or a package version disappears during generation, the command fails. No environment is modified, so there is no system state to roll back at this step.

4. Commit the lockfile

Add conda-lock.yml to version control alongside environment.yml. Review pull requests that touch the lockfile carefully; a changed hash usually means a package was rebuilt or republished.

5. Install from the lockfile

Create the environment using the lockfile instead of conda env create.

# Run in terminal on the target machine
# Requires permission to create conda environments
conda-lock install -n  conda-lock.yml

Replace with the desired environment name. The installer downloads the exact binaries recorded in the lockfile and does not run the conda solver.

Expected check: After installation, run conda list --explicit and confirm the package list matches the lockfile entries for your platform.

Verification

Check that the lockfile matches the manifest

Run the consistency check provided by conda-lock. The exact syntax is version-sensitive; some releases use a flag, others use a subcommand.

# Run in terminal; inspect conda-lock --help for your installed version
conda-lock --check -f environment.yml

If the check fails, regenerate the lockfile and commit the updated file before merging.

Confirm identical installations

Install from the same conda-lock.yml on two machines of the same platform. On each machine, run:

conda list --explicit

Compare the outputs. They should match, including build strings such as py310h1234567_0. Any difference indicates a lockfile mismatch or a platform mismatch.

Limitations

• Platform scope: A lockfile generated for linux-64 does not install on osx-arm64 or win-64. Generate one lockfile per target platform.
• Pip packages are not vendored: conda-lock records pip dependencies, but it does not download or store the wheels. If a pip package is removed from PyPI, installation fails unless you use a private artifact repository or vendor the wheels yourself.
• Conda and pip mixing: Even with a lockfile, conda-installed and pip-installed packages can conflict at runtime. Test the full environment after installation.
• Tool version sensitivity: Lockfiles generated by one conda-lock version may not install cleanly with another. Pin the version in CI.

Rollback

Because conda-lock install changes environment state, rollback means removing the environment and reinstalling from a previous committed version of the lockfile.

# Run in terminal; requires permission to remove conda environments
conda env remove -n 
conda-lock install -n  conda-lock.yml

Check out the previous commit of conda-lock.yml before running the second command.