Diagnosing Docker Image Pull Failures in GitLab CI/CD Pipelines

The Problem: Pipeline Stalls at Image Pull

A GitLab CI/CD pipeline fails before the script section ever executes when the runner cannot retrieve the container image specified in the image: keyword. This typically manifests in the job log as ERROR: Job failed: Failed to pull image or authentication required.

The primary takeaway is that these failures are almost always caused by one of three things: network isolation, invalid credentials, or a mismatch between the requested image tag and what exists in the registry.

Diagnostic Matrix

Symptom in Log	Likely Cause	Diagnostic Check	Primary Fix
Connection timeout / Refused	Firewall or Proxy	curl -I registry URL from runner	Open port 443 or configure daemon.json proxy
"authentication required"	Expired/Missing Credentials	Verify masked CI/CD variables	Update CI_REGISTRY_PASSWORD or add docker login
"manifest unknown"	Incorrect Tag/Image Name	Check Registry UI for existing tags	Correct the image tag in .gitlab-ci.yml
"client is too old" / TLS error	Outdated Docker Client	docker version on runner host	Update Docker Engine on the runner host
HTTP 429 / Too Many Requests	Registry Rate Limiting	Check registry quota/usage	Authenticate pull or use a private mirror

Ordered Troubleshooting Steps

1. Verify Network Reachability

If the job hangs or returns a timeout, the runner host likely cannot reach the registry. SSH into the runner host with sudo permissions and run:

# Replace with your actual registry hostname
curl -I https://registry.example.com

Expected Result: An HTTP 200 or 401 (Unauthorized) response. A timeout indicates a firewall block or a DNS resolution failure. If using a corporate proxy, ensure the Docker daemon is configured via /etc/systemd/system/docker.service.d/http-proxy.conf.

2. Validate Credentials and Authentication

For private registries, the runner needs a valid token. If you are not using the built-in DOCKER_AUTH_CONFIG variable, you must authenticate manually. Test this on the runner host:

# Use placeholders for your variables
docker login [REGISTRY_URL] -u [USERNAME] -p [PASSWORD]

Risk: Never echo passwords in your .gitlab-ci.yml scripts. Always use masked CI/CD variables and pass them via stdin:

before_script:
  - echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin

3. Confirm Image and Tag Existence

A common failure is referencing a tag that hasn't been pushed yet (e.g., using a commit SHA before the build stage completes). Use the GitLab API to verify the tag exists:

# Run from a local terminal with a Personal Access Token
curl --header "PRIVATE-TOKEN: <your_token>" "https://gitlab.com/api/v4/projects/[PROJECT_ID]/registry/repositories/[REPO_ID]/tags"

If the tag is missing, verify your build pipeline's push logic or correct the tag in the consuming job.

4. Inspect Docker Daemon Configuration

If you receive TLS or "insecure registry" errors, check the daemon configuration file on the runner host:

cat /etc/docker/daemon.json

Ensure that any internal registries are listed under "insecure-registries" if they do not support HTTPS. Warning: This is a security risk and should only be used in isolated environments.

Escalation Criteria

Escalate the issue to the following teams if these conditions are met:

• Network/Security Team: If curl fails to reach the registry but the registry is confirmed online.
• Registry Administrator: If docker login fails despite using verified, current credentials.
• Infrastructure/DevOps: If the Docker version on the runner is outdated and the user lacks permissions to run apt-get install or yum update.

Verification and Limitations

To verify the fix, trigger a new pipeline and monitor the Preparing environment phase. A successful pull will show the image being downloaded and the job proceeding immediately to the script section.

Limitations:

• Shared Runners: If using GitLab.com shared runners, you cannot SSH into the host or modify daemon.json. You must rely on DOCKER_AUTH_CONFIG or public images.
• Caching: Some runners cache images. A failure might disappear if the image was manually pulled onto the host, masking a configuration error that will affect other runners.

Example Configuration

This configuration demonstrates a secure pull from a private GitLab registry using masked variables:

image: $CI_REGISTRY_IMAGE:latest

variables:
  # Ensures the docker driver is optimized for CI
  DOCKER_DRIVER: overlay2

build_job:
  stage: build
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA