Reproducible Data Science with Anaconda environment.yml

Why reproducibility matters

When a notebook runs on your laptop but fails on a colleague’s machine or in a CI pipeline, the usual suspect is a mismatch in package versions. Anaconda’s conda tool solves this by letting you declare the exact set of packages you need in a plain‑text environment.yml file. Committing that file to version control gives anyone the ability to recreate the same binary environment with a single command.

What goes into an environment.yml

The file is YAML‑formatted and contains a few top‑level keys:

• name (optional) – the environment name that conda env create will use.
• channels – a list of conda channels, ordered by priority.
• dependencies – packages to install. You can specify just the name (numpy) to get the latest compatible build, or pin a version (numpy=1.24.3).
• pip (under dependencies) – a sub‑section for PyPI packages that are not available on conda channels.

When conda env create -f environment.yml runs, conda performs a single solve step that respects all version constraints and channel order, installing everything in one transaction. This avoids the half‑installed states that can happen when packages are added sequentially with pip.

Worked example: creating a minimal reproducible environment

Follow these steps in a temporary directory (you can delete it afterward). No special privileges are required beyond having conda on your PATH.

1. Create a file named environment.yml with the following content:

name: demo-env
channels:
  - conda-forge
dependencies:
  - python=3.11
  - numpy=1.24.3
  - pandas
  - pip
  - pip:
      - requests==2.31.0

1. Create the environment:

conda env create -f environment.yml

1. Activate it and check the installed versions:

conda activate demo-env
conda list

You should see python 3.11.*, numpy 1.24.3, pandas (latest compatible build from conda‑forge), and requests 2.31.0 listed under the pip section.

1. Deactivate and clean up (this step changes state, so it serves as a rollback):

conda deactivate
conda env remove -n demo-env

Repeating the conda env create command on another machine or in a CI job will produce the same binary packages, assuming the same channel availability.

Trade‑offs and practical limitations

• Channel priority: If a package exists in more than one channel, conda picks the first matching build. Unexpected builds can appear if you rely on the default flexible ordering. Setting channel_priority: strict in your .condarc or adding it directly to environment.yml forces conda to use the exact order you list.
• Solve time: Large environments with many dependencies can take noticeable seconds or minutes to solve because conda evaluates the entire dependency graph at once. Pinning versions (as in the example) reduces the search space and speeds up subsequent solves.
• Lock‑file alternative: For maximum predictability, teams often generate a lock file with tools like conda-lock or mamba lock. The lock file records the exact build strings and hashes, eliminating any ambiguity that could arise from channel updates.

Actionable closing

Start by adding an environment.yml to the root of your project repository. Keep the file small and version‑pinned for packages that are critical to reproducibility, and use the channels section to declare your preferred sources. In CI pipelines, replace ad‑hoc conda install steps with a single conda env create -f environment.yml call, followed by activation. If you notice long solve times, consider generating a lock file and checking it in alongside the environment.yml. This practice gives you and your team confidence that the same environment will run everywhere, reducing the dreaded “works‑on‑my‑machine” bug.