Limits of 409 Conflict as a retry-success signal in PyPI uploads
0 reputation · 29 Sept 2020, 20:22 UTC
0 reputation · 29 Sept 2020, 20:22 UTC
When building an automated release client for PyPI, the goal is to retry distribution uploads after transient network failures without creating duplicate artifacts or masking genuine build errors. PyPI's legacy upload API returns HTTP 409 Conflict when a file already exists, and tools such as Twine interpret this response as a signal to skip the upload. This behavior supports idempotent retries only when the retried artifact is byte-for-byte identical to the original.
The uncertainty arises when a build pipeline regenerates a wheel or source distribution under the same version string. Because PyPI rejects any second upload for that version regardless of content hash, the same 409 response is returned whether the client is retrying an identical file or attempting to publish a modified artifact. Without a server-side retry mechanism or a documented way to query whether a specific file hash already exists, the client cannot reliably distinguish a safe retry from a rejected new upload.
Does PyPI's upload API expose any documented distinction between an exact-duplicate retry and a conflicting new artifact under the same version? Is interpreting 409 as upload success still supported when byte-for-byte reproducibility of the local artifact is not guaranteed? Are there limits to using the JSON simple API's idempotent metadata responses to confirm upload state before deciding whether to retry?
35775 reputation · 30 Sept 2020, 00:49 UTC
No, PyPI's upload API does not expose a documented distinction between an exact-duplicate retry and a conflicting new artifact. A 409 Conflict is returned whenever a file with the same filename already exists for that version, regardless of whether the content hash matches. Interpreting 409 as "success" is only safe if your build pipeline guarantees byte-for-byte reproducibility.
In the PyPI legacy upload API, the 409 Conflict response is a blunt instrument. It signals that the server already possesses a file with the requested name for the specified version. Because PyPI prohibits the overwriting of existing releases to ensure package immutability, the server rejects the upload without performing a content-hash comparison to see if the new file is identical to the old one.
Tools like Twine treat 409 as a signal to skip the upload, effectively treating it as an idempotent success. This assumption holds only if the local artifact is identical to the server's copy. If your pipeline regenerates a wheel (e.g., due to timestamps, build environment changes, or source modifications) but keeps the same version string, the 409 response will mask the fact that the artifact you intended to upload was rejected in favor of a different, existing version.
The JSON Simple API allows you to query existing releases and their associated files, but it is insufficient for verifying artifact integrity. While it provides filenames and sizes, it does not provide a server-side checksum (like SHA-256) for the uploaded files. Therefore, you cannot use the Simple API to confirm that a specific hash has been uploaded before deciding to retry.
To avoid masking build errors or publishing incorrect artifacts, follow these steps:
build with environment pinning to ensure that the same source always produces the same byte-for-byte wheel.Missing Diagnostic: Are you currently using a deterministic build environment (e.g., pinned pip/setuptools versions and SOURCE_DATE_EPOCH)? If not, the 409 signal should be treated as a failure, not a success.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.