Question
404 Not Found for manifest after upstream update in Harbor proxy cache
Dev Orbit
0 reputation · 04 Mar 2022, 14:19 UTC
68.6K views
Loading…
0 reputation · 04 Mar 2022, 14:19 UTC
When an upstream registry updates a mutable tag, Harbor’s proxy cache may still serve a previously cached manifest. The Docker client then receives a 404 Not Found when pulling the new digest, indicating that the proxy cache has not yet been refreshed.
Proxy cache entries are governed by a configurable TTL, and Harbor does not currently perform synchronous invalidation on upstream writes. The core UI and API rely on Redis for metadata caching, which also respects TTLs. This design leaves a trade‑off between consistency (immediate invalidation) and performance (batch TTL expiration).
1. Does Harbor provide any mechanism to trigger immediate cache invalidation for a specific manifest or tag when the upstream content changes?
2. If not, how does the current TTL setting impact the window during which stale manifests may be served, and can this window be reduced without degrading overall cache hit rates?
3. Are there version‑specific differences (e.g., Harbor 2.x vs 3.x) that alter the invalidation behavior for proxy and metadata caches?
33775 reputation · 04 Mar 2022, 22:59 UTC
1. Immediate Invalidation? Harbor does not expose a UI or API endpoint that can evict a specific proxy‑cache entry or tag on demand. The proxy cache is governed solely by a time‑to‑live (TTL) value; entries are purged automatically when the TTL expires.
2. Effect of TTL and Reducing the Window The TTL is configurable (default 24 h in both Harbor 2.x and 3.x). A shorter TTL narrows the period during which a stale manifest can be served, but it also increases the probability that a pull will miss the cache and hit the upstream registry, raising network traffic and potentially slowing the first pull for clients.
3. Version‑Specific Differences Harbor 2.x and Harbor 3.x share the same TTL‑based invalidation logic for the proxy cache and the Redis‑backed metadata cache. No recent release (including the 3.0 series) has introduced synchronous invalidation on upstream writes, nor has the API for artifact refresh been extended to affect proxy entries.
Locate the proxy cache TTL in harbor.yml under the proxy_cache section. It is expressed in seconds; the default is 86400 (24 h).
harborproxy_cache:
ttl: 86400
If you need a tighter consistency window, adjust the value. For example, to serve a 2‑hour window, set ttl: 7200 and restart Harbor to apply the change.
After changing the TTL, monitor cache hit/miss ratios with Harbor’s monitoring endpoints or by inspecting Redis keys: redis-cli KEYS "proxy_cache:*" and redis-cli TTL "proxy_cache:*".
To guarantee immediate consistency (e.g., during a critical rollout), you can flush the entire cache by restarting Harbor or by deleting the Redis keys manually. Note that this will clear all cached manifests, causing a spike in upstream traffic until the cache rebuilds.
To fine‑tune the TTL recommendation for your workload, could you share the current value of proxy_cache.ttl as defined in your harbor.yml? This will help determine whether a moderate reduction (e.g., 12 h) would be acceptable without impacting hit rates.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.