How to reference a requirements.txt in the pyproject.toml of a setuptools project?
Sep 4
I'm trying to migrate a setuptools-based project from the legacy setup.py towards modern pyproject.toml configuration.
At the same time I want to keep well established workflows based on pip-compile, i.e., a requirements.in that gets compiled to a requirements.txt (for end-user / non-library projects of course). This has important benefits as a result of the full transparency:
- 100% reproducible installs due to pinning the full transitive closure of dependencies.
- better understanding of dependency conflicts in the transitive closure of dependencies.
For this reason I don't want to maintain the dependencies directly inside the pyproject.toml via a dependencies = [] list, but rather externally in the pip-compiled managed requirements.txt.
This makes me wonder: Is there a way to reference a requirements.txt file in the pyproject.toml configuration, without having to fallback to a setup.py script?
python1 answer
Accepted answer · original discussion
Sep 4
In setuptools 62.6 the file directive was made available for dependencies and optional-dependencies.
Use dynamic metadata:
[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
Note that the referenced file will use a requirements.txt-like syntax; each line must conform to PEP 508, so flags like -r, -c, and -e are not supported inside this requirements.txt. Also note that this capability is still technically in beta.
Additionally:
If you are using an old version of
setuptools, you might need to ensure that all files referenced by thefiledirective are included in thesdist(you can do that viaMANIFEST.inor using plugins such assetuptools-scm, please have a look on [sic] Controlling files in the distribution for more information).Changed in version 66.1.0: Newer versions of
setuptoolswill automatically add these files to thesdist.
If you want to use optional-dependencies, say, with a requirements-dev.txt, you will need to put an extra group, as follows (credit to Billbottom):
[project]
dynamic = ["dependencies", "optional-dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
optional-dependencies = {dev = { file = ["requirements-dev.txt"] }}
However:
Currently, when specifying
optional-dependenciesdynamically, all of the groups must be specified dynamically; one can not specify some of them statically and some of them dynamically.
2 question comments
Use comments to ask for clarification. Post a solution as an answer.
Nov 9
pyproject.toml. So the question you should ask yourself should be: "how can I get pip-compile (and other packaging tools) to use the list of dependencies from the pyproject.toml as input". Indeed the standardized pyproject.toml should be the source of truth.Nov 9
pyproject.toml should not be equivalent to the typical requirements.in but rather the fully compiled output. Of course, you can copy/paste the compiled output of pip-compile back into the pyprojects.toml but that makes an automated "bump dependency" process awkward. In such cases it is easier to separate the "machine generated" parts from the otherwise human-maintained pyproject.toml content.