Diagnosing Harbor Replication Failures and Image Pull Errors

When a Harbor registry returns a '500 Internal Server Error' during image pushes or leaves replication tasks in a perpetual 'Pending' state, the failure is rarely due to the application logic. These symptoms typically indicate a disconnect between the Harbor core service and its storage backend, the PostgreSQL metadata database, or an upstream proxy timeout. This guide provides a diagnostic path to isolate and resolve these infrastructure bottlenecks.

Identifying the Failure Pattern

Before modifying configurations, match your observed error behavior to the likely root cause to avoid unnecessary restarts.

Symptom	Likely Cause
'500 Internal Server Error' on push/upload	Storage backend (S3/Azure Blob/Local) is unreachable or lacks write permissions.
Replication stuck in 'Pending'	Network firewall blocking traffic to target registry or expired project secrets.
'Manifest not found' during image pull	Race condition between PostgreSQL metadata and storage layers, often from interrupted pushes.
Timeout during large layer downloads	Nginx proxy_read_timeout is shorter than the upstream response time.
Intermittent 'layer missing' errors	Aggressive Garbage Collection (GC) policies deleting active layers.

Step 1: Verifying Storage Backend Connectivity

If Harbor fails during uploads, the registry likely cannot write to the storage provider. Even if the UI shows the storage as connected, specific API requests may hit routing limits or IAM permission denials.

To test this, execute a manual connectivity check from within the Harbor core container. This bypasses the UI and tests the actual network path from the application's perspective.

# Run this on the Harbor host with root/sudo permissions
docker exec -it harbor-core /bin/sh

# For S3 (replace with your actual endpoint and bucket)
curl -I https://<your-bucket>.s3.amazonaws.com/

# For Local storage (verify mount permissions)
ls -ld /var/lib/harbor/data

If the curl command returns a 403 Forbidden or Connection Timeout, verify your IAM policies, security groups, or local Linux file permissions on the data volume. Ensure system clocks are synchronized via NTP, as storage providers like AWS S3 reject requests with significant time drift.

Step 2: Resolving Replication Stalls

Replication tasks that never move past "Pending" usually indicate that the handshake between Harbor and the target registry is failing silently or being blocked by a network appliance.

1. Navigate to Replication in the Harbor UI.
2. Select the failed task and view the Logs.
3. Look for HTTP 401 (Unauthorized) or 403 (Forbidden) status codes.
4. If credentials are the issue, go to Registry Endpoints and re-authenticate with the target registry to refresh the secret token.
5. Verify that the Harbor node has outbound access to the target registry's IP on port 443.

Step 3: Adjusting Proxy Timeouts for Large Images

Large layers (multiple gigabytes) often fail because the Nginx reverse proxy closes the connection before the storage backend finishes streaming the data. If you see upstream timed out in your Nginx error logs, you must increase the timeouts.

Edit your nginx.conf or the relevant configuration map to increase the following values:

# Add or update these in the http or server block
proxy_read_timeout 600s;
proxy_connect_timeout 600s;
proxy_send_timeout 600s;

After making these changes, restart the Nginx container to apply the settings: docker-compose restart nginx-proxy.

Step 4: Auditing Garbage Collection (GC) Policies

If images that were previously available now fail with "layer not found" errors, your Garbage Collection policy may be too aggressive. This occurs when a tag retention policy deletes layers that are still referenced by manifests not yet fully committed to the database.

• Check Maintenance > Garbage Collection to see the last run time.
• Review your Tag Retention rules to ensure they aren't deleting images too quickly.
• Risk Warning: Avoid running GC while heavy push activity is occurring unless you enable "read-only mode" to prevent the deletion of layers currently being uploaded.

Verification and Results

To verify the fix, attempt a push of a known large image (e.g., a heavy ML base image) and monitor the Harbor-core logs: docker logs -f harbor-core. A successful operation will show a 201 Created response for the manifest and no storage error messages in the logs.

Rollback Procedure

If the Nginx timeout changes cause unexpected instability, revert the nginx.conf to the original values and restart the proxy container: docker-compose restart nginx-proxy.