Using Harbor Proxy Cache to Bypass Docker Hub Rate Limits

Problem: Docker Hub Rate Limits

CI/CD pipelines and Kubernetes clusters that pull images directly from Docker Hub often hit the public rate limits imposed on anonymous requests. A shared NAT gateway can quickly exhaust the 100 pulls per 6 hours quota, causing 429 errors and stalled deployments. The solution is to funnel all pulls through a local registry that can cache the images.

How Harbor Proxy Cache Works

A Harbor proxy cache project is a special type of project that is bound to an upstream registry endpoint. When a client issues a docker pull against the proxy project, Harbor checks its local storage for the requested image. If the image is present, Harbor streams it directly to the client. If it is missing, Harbor fetches the image from the upstream registry, stores it locally, and then serves it. Subsequent pulls for the same tag are served from the cache, eliminating external traffic and bypassing the upstream rate limits.

Key benefits:

• Rate‑limit avoidance – only the first pull of a tag consumes the upstream quota.
• Resilience – if Docker Hub is temporarily unreachable, cached images still serve the cluster.
• Governance – cached artifacts are subject to Harbor’s normal controls: RBAC, vulnerability scanning, and retention rules.
• Performance – internal network speeds are typically faster than public internet pulls.

Setting Up a Proxy Cache Project

These steps assume Harbor 2.x. The UI and CLI commands are shown; the CLI requires the harborctl tool and administrator privileges.

1. Create a Registry Endpoint

Navigate to Registries in the Harbor UI and click New Endpoint.

Field	Example Value
Name	docker-hub-endpoint
URL	https://index.docker.io/v1/
Authentication	Optional – provide Docker Hub credentials for higher limits.
Verify SSL	Yes

Alternatively, use the CLI:

harborctl registry create --name docker-hub-endpoint \
  --url https://index.docker.io/v1/ \
  --username YOUR_DOCKERHUB_USER \
  --password YOUR_DOCKERHUB_TOKEN

2. Create the Proxy Cache Project

In the Harbor UI, go to Projects and click New Project.

• Project Name: dockerhub-cache
• Proxy Cache: Select docker-hub-endpoint
• Public/Private: Private (recommended for internal use)
• Scan on Push: Enabled (if Trivy or another scanner is configured)

CLI equivalent:

harborctl project create --name dockerhub-cache \
  --proxy-cache docker-hub-endpoint \
  --public false \
  --scan-on-push true

3. Verify the Cache Works

From a node that can reach Harbor, run:

# First pull – slow, fetched from Docker Hub
sudo docker pull myharbor.example.com/dockerhub-cache/library/nginx:latest

# Second pull – fast, served from Harbor cache
sudo docker pull myharbor.example.com/dockerhub-cache/library/nginx:latest

To confirm Harbor served the second pull from the cache, check the Harbor audit logs or the project’s artifact list. The first pull will show an upstream_fetch event; the second will show a cache_hit event.

Trade‑offs and Limitations

• Mutable Tags – If an upstream tag such as latest is updated, Harbor may continue to serve the cached copy until it re‑validates the upstream. For reproducible deployments, always reference image digests (e.g., nginx@sha256:…).
• Storage Consumption – Every unique image pulled into a proxy cache consumes storage on Harbor’s backend. Configure retention rules (e.g., delete images not pulled in 30 days) to prevent unbounded growth.
• Pull‑Only – Proxy cache projects cannot receive pushes. If you need to push a custom image, create a normal project instead.
• Supported Upstreams – The feature works with any OCI‑compatible registry, including Docker Hub, GitHub Container Registry, and private Artifactory instances. Verify your Harbor version supports the desired upstream type.
• Authentication – If you attach credentials to the endpoint, Harbor will use them for upstream pulls, raising the effective rate limit. If no credentials are provided, anonymous pulls apply.

Practical Checklist

1. Verify Harbor is on a 2.x release that supports proxy cache.
2. Create a registry endpoint pointing to the desired upstream.
3. Create a proxy cache project bound to that endpoint.
4. Pull an image twice to confirm caching.
5. Inspect audit logs or the artifact list for cache_hit entries.
6. Set up a retention policy on the proxy project to manage storage.
7. Use image digests in deployment manifests to avoid stale tags.

Conclusion

Harbor’s proxy cache projects provide a lightweight, policy‑aware way to mitigate Docker Hub rate limits, improve pull performance, and add a resilience layer to your deployment pipeline. By following the steps above, you can quickly set up a cache, verify its operation, and tune retention to match your organization’s storage constraints.