Reproducibility gap across GitLab CI/CD runners A team using GitLab CI/CD wants identical job execution environments across multiple runner hosts so that the same commit produces the same artifacts regardless of which runner picks up the job. The .gitlab-ci.yml image keyword accepts registry tags or Docker-style digests, and include: project can reference another repository by branch, tag, or commit SHA. Registry tags are generally mutable, runner pull_policy and local image caches can serve previously pulled contents, and branch-based include refs advance over time. Digest and SHA pinning improve stability but add update friction and may vary in support across runner versions and executors. Which pinning strategy for job images and included templates reliably eliminates cross-runner drift without assuming registry immutability? How does runner pull_policy interact with cached or mirrored images to mask upstream tag changes for a specific executor? What verification steps confirm that two runners are executing against the same image contents when the pipeline definition is identical?

GitLab CI image pinning and include ref drift across runners

1 answer

35775 reputation · 08 Nov 2022, 07:13 UTC

Answer

Pin job images by Docker digest (image: registry.example.com/project/app@sha256:…) and pin included templates by immutable commit SHA (ref: a1b2c3d…). This guarantees identical image layers and template content across all runners, regardless of tag mutability or branch updates.

Why this works (confirmed facts)

Digest (sha256:…) points to a specific manifest; pulling that manifest always yields the same layer set.
A commit SHA in Git is immutable; tags and branches can move, but a SHA never changes.
GitLab Runner pull_policy: always forces a fresh manifest pull each job, but if the registry returns the same digest the cached layers are reused; if-not-present relies on local cache and will not see upstream tag changes; never uses only pre‑pulled images and is unsuitable for drift prevention.
Registry mirrors/proxies can serve stale tag manifests unless configured with TTL=0 or digest‑only mode.
For Kubernetes executor the runner’s pull_policy is ignored; image pulling is handled by kubelet, which respects its own imagePullPolicy.

Steps to eliminate drift

Replace every image: line in .gitlab-ci.yml with a digest reference, e.g.:
```
image: registry.example.com/myapp@sha256:a1b2c3d4e5f6…
```

For include: sections, use a commit SHA instead of a branch or tag:

include:
  project: 'my-group/templates'
  ref: 'a1b2c3d4e5f6g7h8i9j0'
  file: '/templates/.gitlab-ci.yml'

Set pull_policy: always either globally in config.toml or per‑job (GitLab Runner 16.x+). This ensures the manifest is checked each run.
If you use a registry mirror, verify it is configured to forward digest requests without caching tag responses (TTL=0 or digest‑only).
For Kubernetes executor, configure the pod’s imagePullPolicy: Always in the runner’s [[runners.kubernetes]] section or via image_pull_secrets; the runner’s pull_policy is ignored.

Verification that two runners use the same image

Add a diagnostic step to each job:
```
script:
  - docker inspect --format='{{.Id}}' $CI_IMAGE || crictl inspecti $CI_IMAGE  # for containerd
```
Capture the output as a job artifact and compare the IDs across runs; identical IDs confirm identical layers.
Alternatively, log the digest from the pull command by enabling CI_DEBUG_TRACE=true; the trace shows the exact docker pull or crictl pull command and the resolved digest.
For included templates, echo the resolved SHA inside the included file:
```
script:
  - echo "INCLUDE_SHA=$CI_COMMIT_SHA"
```
Verify the printed SHA matches the pinned commit in the parent pipeline.

Missing diagnostic detail

Are you using the Kubernetes executor? If yes, the runner’s pull_policy is ignored and you must configure kubelet’s imagePullPolicy (or pod annotations) to achieve the same verification.

Your perspective belongs here.

Join the conversation

GitLab CI job image drift symptom: tag pinning vs digest pinning across runners