Reduce External Registry Pulls with Harbor Proxy Cache

Your CI is hitting Docker Hub rate limits

Monday morning CI fails with toomanyrequests: You have reached your pull rate limit. Dozens of jobs pull the same public base images from Docker Hub, and anonymous or authenticated quotas disappear quickly. You do not want to rewrite every Dockerfile or run a full private mirror of every upstream image.

The useful takeaway: Harbor proxy cache projects work as a local pull-through cache. The first pull of a tag fetches the image from the upstream registry and stores its layers in Harbor. Later pulls of the same tag come from local storage. Client commands do not change; you only point the image reference at your Harbor project.

How a proxy cache project behaves

A Harbor project of type Proxy Cache is configured with an upstream registry URL, such as Docker Hub. On a cache miss, Harbor pulls the requested image from upstream, stores the layers, and returns them to the client. On a cache hit, Harbor serves the stored layers without contacting upstream. The cache is tag-based: re-pulling the same tag may return the cached copy unless the cache is refreshed or Harbor revalidates it. Harbor is not a complete mirror; it only caches images someone has actually pulled.

You can configure upstream credentials per proxy project, which helps with private upstream registries or higher rate-limit tiers. Because Harbor becomes part of the pull path, a Harbor outage can block image pulls even when upstream is healthy.

Worked example: cache Docker Hub through Harbor

This example assumes Harbor 2.x or later. Proxy cache reached general availability in later 2.x releases, so verify the exact GA version and configuration options in your release notes before adopting it.

1. In the Harbor UI, create a project and set its type to Proxy Cache. Enter the upstream registry URL and, if needed, upstream credentials for authenticated pulls.
2. On a workstation or CI runner that can reach Harbor, log in and pull an image through the project:

# Run on a workstation or CI runner with Docker access to Harbor
# Required permissions: Harbor account able to create or administer projects
docker login 

# First pull: Harbor fetches from upstream and caches layers locally
docker pull //library/ubuntu:22.04

# Second pull: served from Harbor cache, not upstream
docker pull //library/ubuntu:22.04

Replace <HARBOR_HOST> with your Harbor hostname and <PROJECT> with the proxy cache project name. Confirm the exact repository path format for your Harbor version in the UI; it typically mirrors the upstream namespace under the project.

Expected checks:

• The Harbor UI lists the project type as Proxy Cache.
• The repository appears under the project after the first pull.
• The second pull completes without new traffic to Docker Hub, visible in Harbor audit logs or your network logs.

Relevant risks: incorrect upstream credentials cause authentication failures, large or numerous images grow Harbor storage, and Harbor downtime blocks pulls that would otherwise succeed directly against upstream.

To verify freshness, compare image digests:

docker inspect --format='{{index .RepoDigests 0}}' //library/ubuntu:22.04
docker inspect --format='{{index .RepoDigests 0}}' docker.io/library/ubuntu:22.04

If the digests differ, the cached tag is stale relative to upstream. Pin security-critical images by digest (ubuntu@sha256:...) rather than by tag alone.

Trade-offs and operational limits

Concern	Practical effect
Storage growth	Cached layers accumulate unless paired with Harbor retention policies or garbage collection.
Stale tags	A cached tag can mask an upstream security fix; tag immutability differs from the upstream source of truth.
Dependency on Harbor	Without a highly available Harbor deployment, the cache turns an upstream outage into a local outage.
Upstream terms	Some registries restrict proxying or rate-limit proxy clients; confirm terms of service before caching public images.

Check storage growth in the Harbor UI under project storage usage, or through the Harbor API, and set a retention policy that matches your recovery needs.

Start small and monitor

Create one proxy cache project for Docker Hub and route a single non-critical CI pipeline through it. Watch pull success rates and Harbor disk usage for two weeks before expanding to more projects or teams. Enable Harbor garbage collection and retention, and treat the cache as a performance shield, not a backup of upstream images.