Why Your Conda Environments Drift and How a Lock File Fixes It

Why Your Conda Environments Drift and How a Lock File Fixes It

When you share a Python project with a teammate or move it to a CI server, the environment you built on your laptop often behaves differently on the other machine. Packages get upgraded, channel order changes, and compiled binaries break. The result is "it works on my machine" and wasted debugging time.

The Problem of Environment Drift

Developers typically export a list of installed packages with conda list or pip freeze and hand it over. That list describes package names but not the exact versions or the channel URLs from which they were fetched. If the default channels are updated, conda install may pull newer releases, causing subtle incompatibilities. This phenomenon is called environment drift.

Enter the Conda Lock File

Conda can automatically capture the exact package versions and the channel URLs used to create an environment. Since conda 23.9, the conda lock command (or conda env export --no-builds) writes a deterministic specification to an environment.yml file. This file can be fed back to conda env create on any machine, guaranteeing that the same versions and channels are used.

Worked Example: Export, Recreate, Verify

1. Create a fresh environment (you need conda installed and write permission in the target directory):

   conda create -n myenv python=3.11 -y

2. Activate the environment:

   conda activate myenv

3. Install the packages you need, for example:

   conda install numpy pandas scipy -y

4. Export the exact specification without build hashes (the lock file):

   conda env export --no-builds > environment-lock.yml

   Here environment-lock.yml is the placeholder for the lock file path. The command writes a YAML document that lists each package, its version, and the channel URL.

5. Recreate the environment elsewhere (requires conda access and sufficient disk space):

   conda env create -f environment-lock.yml -n recreated

6. Verify the match:
   1. Activate the new environment:

      conda activate recreated

   2. List the packages and compare with the lock file:

      conda list --explicit > actual.txt

   3. Open environment-lock.yml and extract the specs: section; compare each line with actual.txt. Any missing or version‑different entry signals a mismatch.

If all lines line up, the lock file successfully reproduced the original environment.

Trade‑off and Limitation

The lock file captures pure Python‑level dependencies and channel metadata, but it does not lock compiled binary packages such as numpy wheels that contain C extensions. Those binaries are selected based on the platform and the channel’s binary availability, so a different OS or a channel that provides a newer binary may still cause incompatibility. Moreover, if you manually edit the environment after export, the lock file becomes stale, leading to version mismatches. Channel priority order remains critical; a package may be found on a lower‑priority channel if the higher‑priority one does not contain the required version.

Actionable Closing

To make reproducible environments a habit, add the lock‑file step to your project’s setup script and commit the generated environment-lock.yml to version control. In CI pipelines, replace manual conda install commands with conda env create -f environment-lock.yml. Periodically re‑export the lock file after a deliberate change (e.g., adding a new dependency) and run the verification steps above. By treating the lock file as the single source of truth for package versions and channels, you eliminate most environment drift while staying aware of its binary‑compatibility caveats.