Choosing the Right Harbor Replication Strategy for Multi‑Site Consistency

Decision & Constraints

When you run Harbor in multiple data centers or cloud regions, you must keep container images in sync. The core decision is which replication mode best balances Recovery Point Objective (RPO), bandwidth budgets, security, and operator overhead. Harbor offers three native replication modes that can be combined with filter rules and retention policies:

• Scheduled Pull – the destination pulls from the source on a cron schedule.
• Webhook Push – the source pushes to the destination when an image is pushed.
• Proxy Cache – the destination pulls on demand and caches layers.

Each mode has different latency, bandwidth, and security footprints. The decision must also account for:

• RPO: how fresh must the destination be after a new image is pushed?
• Bandwidth: limited outbound or inbound capacity.
• Trust: whether the two registries can open inbound connections or must rely on outbound traffic.
• Audit & immutability: whether you need an immutable audit trail for compliance.
• Operator effort: credential management, endpoint configuration, and monitoring.

Options Comparison

Mode	Latency (RPO)	Bandwidth Use	Security Model	Operator Overhead	Storage Impact
Scheduled Pull	Up to schedule interval (e.g., 5‑30 min)	Peak during sync, can be throttled	Outbound only, works behind firewalls	Low – simple endpoint config, cron job	Full copy of images
Webhook Push	Near‑real‑time (seconds)	Continuous, but only for new images	Inbound connection required; event delivery must be reliable	High – need to expose webhook endpoint, monitor retries	Full copy of images
Proxy Cache	First request latency (up to upstream latency)	On‑demand; no extra bandwidth for unrequested images	Outbound only; no inbound traffic	Medium – configure caching policy, monitor cache size	Only cached layers, reduces storage

Trade‑Off Analysis

1. Latency vs. Bandwidth
   • Webhook push gives the lowest RPO but consumes bandwidth continuously for each new image.
   • Scheduled pull can throttle bandwidth but introduces a delay equal to the schedule.
   • Proxy cache keeps bandwidth low but cannot guarantee an image is present until it’s requested.
2. Security & Trust
   • Pull mode is ideal when the destination cannot accept inbound traffic or when you rely on outbound-only connectivity.
   • Push mode requires the destination to expose a webhook endpoint and trust the source’s TLS certificate.
   • Proxy cache only needs outbound connectivity, which is often easier to secure.
3. Operator Effort
   • Push mode demands continuous monitoring of webhook health and retry logic.
   • Pull mode is largely declarative: set a schedule and let Harbor handle it.
   • Proxy cache requires tuning of cache eviction and storage limits.
4. Audit & Retention
   • All modes preserve image digests and metadata, but push mode can expose the source’s push history in the destination’s audit logs.
   • Retention policies must be applied at the destination; deleting an image at the source does not automatically delete it at the destination unless a delete replication rule is enabled.

Concrete Validation Workflow

Below is a step‑by‑step example that demonstrates creating a Scheduled Pull policy, verifying that it respects filter rules, and confirming that image digests match. The same steps can be adapted for the other modes by changing the mode field in the policy JSON.

1. Prerequisites
   • Harbor v2.9+ (check harbor version).
   • Admin credentials for both source and destination registries.
   • Network connectivity from destination to source (HTTPS on port 443).
2. Create a Replication Endpoint

   # POST /api/v2.0/replication/endpoints
   curl -k -u admin:adminpass \
        -H "Content-Type: application/json" \
        -d '{"name": "source-endpoint", "type": "harbor", "url": "https://source-harbor.example.com", "credential": {"username": "replicator", "password": "s3cr3t"}, "ca": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"}' \
        https://destination-harbor.example.com/api/v2.0/replication/endpoints
   

   • Replace placeholders with actual values.
   • Verify the response contains "id": 1 indicating the endpoint was created.
3. Create a Replication Policy

   # POST /api/v2.0/replication/policies
   curl -k -u admin:adminpass \
        -H "Content-Type: application/json" \
        -d '{"name": "prod-pull", "sourceProjectID": 1, "targetEndpointID": 1, "targetProjectID": 2, "filters": {"repositories": ["app/*"], "tags": ["^v[0-9]+\.[0-9]+\.[0-9]+$"]}, "mode": "pull", "schedule": "0 */5 * * *", "enabled": true}' \
        https://destination-harbor.example.com/api/v2.0/replication/policies
   

   • The schedule uses a cron expression that runs every 5 minutes.
   • The filter keeps only repositories matching app/* and tags that look like semantic versions.
4. Trigger a Test Push

   # Tag and push an image to the source project
   docker tag alpine:latest source-harbor.example.com/project/app:v1.0.0
   docker push source-harbor.example.com/project/app:v1.0.0
   

5. Verify Replication Job

   # List recent jobs
   curl -k -u admin:adminpass https://destination-harbor.example.com/api/v2.0/replication/jobs | jq '.[0]'
   

   • Check the status field is Success and that the sourceDigest matches the targetDigest.
   • Inspect the message for any layer transfer errors.
6. Manual Digest Comparison

   # Source digest
   curl -k -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
        https://source-harbor.example.com/v2/project/app/manifests/v1.0.0 | jq -r '.config.digest'
   # Destination digest
   curl -k -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
        https://destination-harbor.example.com/v2/project/app/manifests/v1.0.0 | jq -r '.config.digest'
   

   • Both digests should be identical, confirming the image was replicated intact.

Limitations & Practical Checks

• Harbor replication APIs and UI behavior can differ between minor releases. Verify that mode values are supported in your version.
• Replication does not automatically propagate deletions unless you enable the delete flag in the policy.
• Large layer sizes can saturate bandwidth; consider using maxBandwidth in the endpoint configuration if available.
• Always monitor job history for failures; silent failures can occur if credentials expire.
• Use harborctl or the UI to check that the destination project’s retention policy is applied after replication.

Conclusion

For most environments where outbound connectivity is reliable and the primary goal is to keep a full copy of images with a predictable RPO, Scheduled Pull is the simplest, most secure, and least operator‑heavy choice. If you need near‑real‑time propagation and can expose a webhook endpoint, Webhook Push delivers lower latency at the cost of increased coupling. When bandwidth is scarce and you only need on‑demand availability, Proxy Cache reduces storage and traffic but sacrifices guaranteed presence.

By following the validation workflow above, you can confirm that your chosen replication strategy meets the defined constraints and that images remain consistent across sites.