GitLab CI/CD Cache Trust Boundaries and Storage Operations

Requirements for Reliable GitLab CI/CD Cache

GitLab CI/CD cache accelerates pipeline execution by reusing dependencies and build artifacts across jobs. Before designing or troubleshooting cache behavior, three baseline requirements must be satisfied:

1. Runner capability: Runners must be registered with a token that grants access to the target project's CI/CD configuration. Shared or group runners require explicit project membership validation.
2. Storage durability: The underlying filesystem or object store must retain directory entries until the configured artifact_expiration elapses or manual invalidation occurs.
3. Visibility enforcement: GitLab's server must verify that a cache key's project membership and project visibility (public, internal, private) match the requesting runner's permissions before returning data.

If any of these are misaligned--for example, a runner without project access attempting to pull a private project's cache--GitLab returns a 401 or 404 response, and the job fails with a cache miss.

Smallest Suitable Design

The minimal setup for functional cache in GitLab consists of a single runner attached to a project, with default cache storage under the GitLab instance's shared directory. No additional configuration is required for basic use cases. However, for multi-tenant environments or pipelines that span multiple runners, the smallest suitable design adds:

1. A runner registration token scoped to the project (or group) that needs cache access.
2. A defined cache section in .gitlab-ci.yml with an explicit key so that cache isolation is deterministic.
3. A retention policy set via artifact_expiration in the project's CI/CD settings or pipeline YAML, preventing indefinite accumulation.

This design keeps cache co-located with the GitLab instance, avoiding external dependencies while still providing key-based isolation.

Trust and Data Boundaries

GitLab enforces cache trust boundaries at two levels: the runner registration layer and the project-level CI/CD settings.

Runner registration tokens bind a runner to a specific project or group. A runner registered at the group level can access cache for projects within that group, but cannot access cache for projects outside the group unless explicitly granted. Registration tokens are validated during the gitlab-runner verify handshake; mismatched tokens result in immediate cache rejection.

Project-level CI/CD settings include cache key patterns and artifact_expiration values. The GitLab server checks each incoming cache request against the project's visibility flag. A private project's cache is inaccessible to runners belonging to different projects or to unauthenticated runners. This check happens before the server streams cache contents back to the runner, ensuring that no data leaks across trust boundaries.

For teams using protected or masked variables in combination with cache, GitLab also validates that the job has permission to read or write the specified key path. This prevents a job from inadvertently overwriting another project's cache entry through a poorly scoped key.

Operational Checks and Monitoring

Operational visibility into cache health relies on GitLab's built-in Prometheus metrics and UI sections. The most relevant metrics include:

• gitlab_ci_cache_hits_total and gitlab_ci_cache_misses_total--cumulative counts of cache hits and misses across all runners.
• gitlab_ci_cache_size_bytes--current disk usage attributable to cache directories per project.

These metrics are exposed under the GitLab Prometheus endpoint (typically /metrics) and can be fed into alerting rules. A practical alert pattern triggers when gitlab_ci_cache_size_bytes exceeds 80 percent of the allocated storage quota, or when the hit rate drops below 70 percent over a seven-day window, suggesting misconfigured keys or runner misalignment.

In the GitLab UI, navigate to Project > CI/CD > Artifacts for a specific pipeline. The Cache section lists expected keys, their sizes, and retention dates matching the project's artifact_expiration setting. This is the fastest way to verify that a given pipeline job populated or consumed cache as intended.

To cross-reference runner behavior, run a pipeline job with a defined cache key and inspect the job output for the phrases Cache hit or Cache miss. The runner version and GitLab instance version must be compatible; major version gaps can change cache serialization formats, as noted in the cautions section.

Failure Modes and Diagnostic Decision Points

When cache storage becomes saturated, pipelines exhibit specific failure patterns:

• Inode or disk-quota exhaustion: GitLab returns 404 Cache or artifact not found responses. The job logs show cache_key: key not found even though the key was previously used.
• Partial cache uploads: Network interruptions during cache upload leave orphaned directory entries. On retry, the runner may report a cache miss or, in rare cases, corrupt the existing entry.
• Misconfigured artifact_expiration: If set to never or a very large value, cache entries accumulate indefinitely. This inflates storage costs and can degrade instance performance, especially on installations using PostgreSQL-backed storage metadata.

Diagnosing these issues begins with checking the Prometheus metrics mentioned above, then inspecting the relevant project's disk usage via the instance's filesystem or cloud provider console. If the 404 Cache error persists after freeing space, verify that the cache key pattern in .gitlab-ci.yml matches the project's visibility and runner registration scope.

Conditions That Would Change the Design

While the local-filecache design suffices for many teams, several conditions prompt a re-architecture toward object-storage backends such as AWS S3, MinIO, or Google Cloud Storage:

• Cross-runner cache sharing without a shared filesystem: When runners operate on isolated hosts without NFS or a mounted volume, cache entries are siloed. Migrating to an object store makes cache globally available to all runners that can authenticate to the bucket.
• Compliance-driven immutability: Regulations such as SOX or GDPR may require immutable artifact retention. Object storage backends can enforce Object Lock or similar policies, whereas local filesystems cannot.
• Database shard decoupling: In large GitLab instances, cache metadata stored in the primary PostgreSQL shard can become a bottleneck. Offloading cache blobs to an external store reduces load on the database and improves pipeline scheduling latency.
• Storage cost optimization: If instance disk usage consistently approaches quota and archiving old pipelines is undesirable, moving cache to cheaper cold-storage tiers becomes a practical lever.

Migrating to an object store requires updating the GitLab runner configuration with cache_dir pointing to an S3-compatible endpoint, and setting GITLAB_CI environment variables or instance-level settings for artifact_storage. The migration path includes a gitlab-rake gitlab:cache:migrate utility (available in GitLab 16.x+) that rewrites local cache entries to the new backend, but teams should validate the process on a staging runner before applying it to production.

Cautions and Version Considerations

Several operational gotchas merit attention:

• Misconfigured artifact_expiration: Leaving expiration unset or set to never causes cache to persist indefinitely. Review project-level CI/CD settings quarterly and align expiration with your data-retention policy.
• Network interruptions during cache upload: If a runner loses connectivity while uploading cache, GitLab may record a partial entry. The next job run reports a cache miss, and the runner retries the upload. Add retry logic at the runner level if flaky network conditions are expected.
• Major version upgrades: Upgrading GitLab or GitLab Runner to a major release may change cache serialization formats. The research cautions that manual cache clearing or migration scripts may be required after upgrading from GitLab 15.x to 16.x, or from Runner 15.x to 16.x. Always consult the version-specific upgrade guide and perform a test-run before applying to production runners.

These cautions are not exhaustive but represent the most common failure points observed in GitLab operations.

Verification Checklist

1. In the GitLab UI, go to Project > CI/CD > Artifacts for a pipeline and verify the Cache section lists expected keys and retention dates matching the project's artifact_expiration.
2. Execute a pipeline job with a defined cache key (for example, key: build-{{ .Branch }}) and inspect the job output for Cache hit or Cache miss indicators.
3. Cross-reference the GitLab instance version and runner version listed in the job metadata; ensure they are within a compatible minor-version range.
4. Check Prometheus metrics gitlab_ci_cache_hits_total and gitlab_ci_cache_misses_total for the expected trend; investigate sudden miss spikes.
5. Run df -h on the GitLab server host and du -sh /var/opt/gitlab/gitlab-rails/shared/artifacts (path may vary by installation) to confirm disk usage aligns with metric reports.

If any step reveals a mismatch, consult the version-specific upgrade notes or consider the migration conditions described earlier.