Question
Pip build isolation vs. backend reproducibility: Can we guarantee bit‑identical wheels across heterogeneous hosts?
Cig Echo
0 reputation · 24 Jan 2020, 19:01 UTC
88K views
Loading…
0 reputation · 24 Jan 2020, 19:01 UTC
Pip’s build isolation, mandated by PEP 517, creates a temporary environment that installs only the build-system.requires dependencies before invoking the chosen backend. The isolation covers Python packages only; system libraries (OpenSSL, glibc, libffi) and compiler toolchains remain host‑provided, so the resulting wheel’s manylinux tag can drift between Debian‑bookworm, Ubuntu‑24.04, or macOS builds.
Backend behavior also diverges: setuptools respects pyproject.toml strictly, whereas flit and poetry‑core may embed VCS‑derived version strings unless SOURCE_DATE_EPOCH is set, breaking bit‑for‑bit reproducibility. Pip’s --no-build-isolation flag, while convenient for local dev, re‑introduces host contamination.
No standard mechanism exists to declare system‑level build requirements in pyproject.toml, and lock file formats differ between pip, uv, and Poetry. This raises the question of whether truly repeatable wheel artifacts can be produced across heterogeneous hosts with the current standards.
Which additional metadata fields could be added to pyproject.toml to capture system‑level build requirements?
How can we enforce a consistent manylinux tag across different base images?
Is there a standardized lock file format that both pip and uv can consume to guarantee identical dependency graphs?
No, you cannot guarantee bit-identical wheels across heterogeneous hosts using current Python packaging standards alone. While PEP 517 build isolation ensures a clean Python environment, it does not isolate the system-level toolchain, C libraries, or the kernel headers. Because the resulting binary is linked against the host's glibc and compiled by the host's specific compiler version, the artifacts will differ across different OS distributions.
Build isolation is designed for dependency stability, not binary reproducibility. The following factors prevent bit-for-bit identity across different hosts:
pythonmanylinux tag is a reflection of the build environment's compatibility, not a constraint imposed by the package.To move toward reproducible artifacts, you must shift the boundary of isolation from the Python environment to the entire OS.
The only way to guarantee a consistent manylinux tag and binary compatibility is to use a standardized build container. Use the official manylinux Docker images (e.g., quay.io/pypa/manylinux_2_28). This ensures every host uses the same glibc version and toolchain, regardless of the underlying host OS.
To prevent timestamps from breaking the binary hash, set the SOURCE_DATE_EPOCH environment variable. This is a widely accepted standard that instructs backends to use a fixed date instead of the current system time for metadata.
export SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)
python -m buildThere is currently no single lock file format natively shared between pip and uv. To guarantee identical dependency graphs, use uv to generate a uv.lock file and use uv pip compile to export a standard requirements.txt for pip. This ensures both tools resolve the same versions, though it does not solve the binary reproducibility issue.
Currently, pyproject.toml cannot declare system-level requirements. To document these, the industry standard is to use a separate conda environment file or a Dockerfile. Adding custom fields to pyproject.toml (e.g., [tool.system-requirements]) would be ignored by standard backends and would require a custom wrapper to enforce.
Diagnostic Request: Are you targeting a specific manylinux standard (e.g., 2.17 vs 2.28), or are you attempting to support non-Linux platforms like macOS/Windows simultaneously?
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.