Question
conda and pip interoperability in Anaconda environments: dependency ownership and metadata consistency
Mira Lane
0 reputation · 17 Jul 2024, 16:22 UTC
78K views1
Loading…
0 reputation · 17 Jul 2024, 16:22 UTC
An Anaconda environment is intended to be reproducible when both conda and pip are used as supported installers for the same prefix. The integration boundary between the two tools is documented, but the ownership model for shared dependencies remains ambiguous.
Conda maintains its own package metadata and pip maintains its own. The solvers operate independently, so a pip install can change a library that conda placed, and conda metadata does not record libraries installed by pip. Official guidance recommends installing as much as possible with conda first and using pip only for packages unavailable from conda channels, but the ordering is advisory rather than enforced.
With no single supported reconciliation mechanism, the environment can diverge between what conda reports and what is actually importable.
33775 reputation · 17 Jul 2024, 17:18 UTC
There is no supported mechanism to "declare" ownership of a dependency when both installers are used; ownership is determined strictly by the tool that performed the installation. If a package is installed via conda, it is owned by the conda solver. If it is installed via pip, it is treated as a foreign entity. When a package exists in both repositories, installing it via pip will overwrite the files in the environment prefix, but it will not update conda's internal metadata database.
When executing conda env export, pip-installed packages are listed under a dedicated - pip: subsection within the YAML file. However, this representation is a flat list of packages and versions, which is insufficient for full reproducibility due to the following missing metadata:
There is no supported method to force conda metadata to reflect changes made by pip without recreating the environment. Conda's solver does not scan the site-packages directory to "adopt" pip-installed versions into its own state.
To maintain the highest possible consistency, follow this sequence:
anacondaconda install.pip install as the final step.You can verify metadata inconsistency by installing a package with conda and then upgrading it via pip:
conda install numpy
pip install numpy --upgrade
conda list numpy
pip list | grep numpyIn this scenario, conda list will often report the original version it installed, while pip list (and the actual Python import) will reflect the upgraded version.
Use comments to ask for clarification. Post a solution as an answer.
33,775 reputation · 17 Jul 2024, 20:09 UTC
When you run conda env export, conda lists pip‑installed packages under a pip: subsection but does not store the exact pip install command or the wheel/build hash used. Therefore, to rebuild the environment identically you need to pair the exported YAML with a pip freeze output (or pip list --format=freeze) and reinstall pip packages after restoring the conda specifications. This ensures that any pip‑only version or build is reproduced without relying on conda’s solver, which ignores pip packages during dependency resolution.