Harbor resumable upload session idempotency and duplicate writes

Answer to the Core Question

Harbor implements the Docker Registry HTTP API v2 resumable upload logic exactly as defined by the upstream Docker Registry code base. The spec guarantees that a PUT to the same session ID (the UUID returned in the Location header) and the same offset is idempotent: the server simply acknowledges the request and does not append the chunk again to the underlying storage. Therefore, duplicate chunks are not written before the final commit, and the eventual content‑addressable deduplication step is performed only after the upload is completed (the final PUT that supplies the digest).

Key Points

• Idempotent handling of PUT with identical session ID & offset – guaranteed by the Docker Registry spec and honored by Harbor in all supported releases.
• During an active upload, data is stored in a temporary, non‑deduplicated blob (typically under uploads/ in the chosen storage driver). Duplicate chunks are not appended again.
• Deduplication occurs only after the final PUT that includes the content digest.
• Incomplete uploads are cleaned up by a background job that runs hourly by default. The interval is driven by the storage delete timeout setting in the registry configuration (e.g., storage.delete.timeout or registry.storage.delete.timeout depending on the release).
• Harbor itself does not expose a dedicated API to change this cleanup interval; you must edit the registry’s config.yml or the Helm values if you’re using the chart.
• Older Harbor releases (pre‑v2.0) may not fully honor the idempotent semantics; clients should test the specific version in use.

Verification Steps (Safe, Scoped Commands)

1. Start a resumable upload:

   curl -v -X POST https://harbor.example.com/v2/myrepo/blobs/uploads/ \
     -H "Content-Type: application/octet-stream"
   # Note the Location header, e.g. https://harbor.example.com/v2/myrepo/blobs/uploads/abcd1234
   

2. Upload a 1 MiB chunk at offset 0:

   curl -v -X PUT https://harbor.example.com/v2/myrepo/blobs/uploads/abcd1234?offset=0 \
     -H "Content-Type: application/octet-stream" \
     --data-binary @chunk1.bin
   # Expect HTTP 308 and a Location header with the same session ID.
   

3. Retry the same chunk (same offset):

   curl -v -X PUT https://harbor.example.com/v2/myrepo/blobs/uploads/abcd1234?offset=0 \
     -H "Content-Type: application/octet-stream" \
     --data-binary @chunk1.bin
   # Response should still be 308. Inspect the storage backend (e.g., list files in uploads/) to confirm only one copy exists.
   

4. Complete the upload by providing the digest:

   curl -v -X PUT https://harbor.example.com/v2/myrepo/blobs/uploads/abcd1234?digest=sha256:abc123... \
     -H "Content-Type: application/octet-stream"
   # Expect HTTP 201 and the final blob available.
   

5. Check cleanup configuration:

   grep -E "storage.delete.timeout|registry.storage.delete.timeout" /etc/docker/registry/config.yml
   # Or, if using Helm, inspect the values.yaml for the same key.
   

What to Ask Next

To tailor the recommendation to your environment, could you share the Harbor version you are running? This will confirm whether the behavior described above applies to your release.

Diagram

Phase	Action	Storage Effect
Start Upload	POST /blobs/uploads/	Allocate temp blob
Chunk PUT	PUT /blobs/uploads/?offset=...	Append data once
Duplicate PUT	Same offset	No extra write
Cleanup	Hourly job	Remove orphaned temp blobs