Solving Dependency Hell: Using Anaconda to Isolate Binary and Python Libraries

The Conflict of Global Site-Packages

When working on multiple data science projects, installing libraries globally into your system's Python site-packages (the default directory where Python stores installed modules) creates a fragile environment. Project A might require NumPy 1.21 for a legacy model, while Project B requires NumPy 1.26 for new features. Installing one overwrites the other, breaking your previous work.

The solution is strict environment isolation. By using Anaconda's Conda manager, you create separate, siloed directories for every project. This ensures that changes in one environment cannot leak into another, allowing you to maintain multiple Python versions and library sets on a single machine.

Beyond Python: Managing Binary Dependencies

A common misconception is that Conda is simply a wrapper for pip. While pip manages Python packages, many data science libraries rely on non-Python binary dependencies: C++ libraries, CUDA toolkits for GPU acceleration, or Intel's Math Kernel Library (MKL).

When you install a package via pip, it assumes the necessary system-level binaries are already installed on your OS. If they aren't, the installation can fail with cryptic compilation errors. Conda treats these binaries as first-class packages. When you install a library like PyTorch or TensorFlow via Conda, it bundles the required CUDA toolkit and C libraries, making the environment portable and self-contained regardless of the host OS configuration.

Workflow for Reproducible Environments

To move a project from a local machine to a production server or a teammate's laptop, you cannot rely on manual installation steps. Instead, use an environment.yml file. This file defines the exact versions of every dependency, including the Python interpreter itself.

The standard workflow involves creating the environment, installing the necessary tools, and then exporting the state to a YAML file. This allows others to recreate your exact environment with a single command, eliminating the "it works on my machine" problem.

Worked Example: Creating and Exporting 'ml-env'

This example shows how to set up a dedicated environment for a machine learning task and share it. Run these commands in your terminal (Linux/macOS) or Anaconda Prompt (Windows). You need Anaconda or Miniconda installed; no elevated permissions are required since environments live in your user directory.

1. Create the environment with specific versions:

# Create an environment named 'ml-env' with Python 3.10, NumPy 1.23, and Pandas 1.5
conda create --name ml-env python=3.10 numpy=1.23 pandas=1.5

2. Activate the environment:

# Switch the current shell context to the new environment
conda activate ml-env

3. Verify the isolation:

# List packages to confirm the specific versions are installed
conda list

Check that the output shows numpy 1.23.x and pandas 1.5.x. If the versions differ, the solver may have adjusted them to resolve a conflict; re-run the create command and read the solver's proposed changes before accepting.

4. Export the environment for a teammate:

# Save the environment specification to a file
conda env export > environment.yml

5. Recreate the environment (for the teammate):

# Create a new environment based on the provided YAML file
conda env create -f environment.yml

Risk note: Avoid mixing conda install and pip install within the same environment whenever possible. Conda cannot track dependencies installed by pip, which can lead to pip overwriting a Conda-managed binary and potentially corrupting the environment. If you must use pip, install Conda packages first and pip packages last.

The Trade-off: Stability vs. Disk Space

The primary cost of this isolation is disk space. Because each environment is a separate directory, duplicating large libraries (like PyTorch or TensorFlow) across five different projects can consume gigabytes of storage quickly.

Conda mitigates this with hard links pointing to a single shared package cache, but metadata and some files are still duplicated per environment. To reclaim space, periodically remove unused packages and cached tarballs:

conda clean --all

This only deletes cached downloads and unreferenced packages, not your active environments, so it is safe to run. If you want to remove an environment you no longer need, use conda env remove --name ml-env.

Verification and Closing

To confirm your isolation is working, run conda env list. You should see all named environments and the paths to their directories. Activate ml-env and run python --version; it should reflect the version you specified at creation, regardless of your system's global Python. As a final reproducibility check, create a fresh environment from your exported environment.yml on another machine and confirm conda list matches.

By treating environments as disposable, version-controlled assets rather than permanent system installations, you remove the risk of dependency conflicts and make your data science pipeline reproducible across machines.