Speeding Up Monorepo Builds with GitLab CI/CD Caching

Problem: repeated dependency downloads in a monorepo

In a large monorepo each service often declares its own dependencies (e.g., package-lock.json, pom.xml, requirements.txt). When a pipeline runs, every job that needs those dependencies pulls them from the remote registry again, even if nothing changed since the last run. This repeats work, consumes CI minutes, and slows feedback for developers.

How GitLab CI/CD caching works

GitLab Runner can store a tarball of specified directories between jobs. At the start of a job the runner attempts to download a cache that matches a user‑defined key; at the end of the job it uploads (or updates) the cache with the same key. The cache is stored on the runner’s local disk by default, but GitLab also offers a centralized cache storage option for shared runners. The cache is immutable during a job – any changes a job makes to cached files are not persisted unless the job pushes a new cache.

Designing a cache key for a monorepo

A good cache key must change whenever the underlying dependencies change, but stay stable when they do not. A common pattern combines:

• the branch or tag reference (${CI_COMMIT_REF_SLUG}) to keep caches separate per line of work,
• the project path (${CI_PROJECT_PATH}) to avoid collisions across projects in the same GitLab instance, and
• a hash of the lock‑file (or a set of lock‑files) that uniquely identifies the exact dependency tree.

For a monorepo you can further prefix the key with a directory identifier so that each service gets its own cache, e.g. services/auth-${hash(services/auth/package-lock.json)}. This keeps storage efficient because unchanged services reuse the same cache while a change in one service only invalidates its own cache.

Worked example: caching Node.js modules

The following .gitlab-ci.yml defines a cache for a Node.js service located in services/web. The key uses the branch name, project path, and a hash of the lock‑file. The job first restores the cache, runs npm ci (which will be a no‑op if the cache hit), then executes the test suite.

variables:
  NODE_ENV: "test"

cache:
  key: "${CI_COMMIT_REF_SLUG}-${CI_PROJECT_PATH}-${hash(services/web/package-lock.json)}"
  paths:
    - services/web/node_modules/

services_web_test:
  stage: test
  script:
    - cd services/web
    - npm ci
    - npm test

When the pipeline runs, the job log will show messages like:

• "Restoring cache..." followed by either "Cache downloaded!" (hit) or "No cache to download" (miss).
• At the end, "Uploading cache..." if the cache was created or updated.

Trade‑offs and limits

While caching can cut minutes from each pipeline, there are practical considerations:

• Staleness: If the key does not include a file that influences the dependencies (e.g., a .npmrc change), the cache may be reused incorrectly, leading to nondeterministic builds.
• Storage quota: GitLab enforces a per‑project cache size limit (default 1 GB on SaaS). Exceeding this causes upload failures and forces a fresh download. Monitoring cache size and periodically pruning old keys is necessary.
• Cache immutability during a job: Any modifications a job makes to cached directories are lost unless the job explicitly pushes a new cache at the end.

Checking that the cache is helping

1. Run a pipeline on a branch with no prior cache; note the duration of the services_web_test job.
2. Push a second commit that does not alter services/web/package-lock.json. The job should show a cache download and the duration should drop noticeably.
3. Alter the lock‑file (e.g., add a dev dependency), push a new commit, and verify the pipeline reports a cache miss and reinstalls dependencies.
4. Visit CI/CD → Cache in the project settings (or use the API endpoint GET /projects/:id/cache) to view current cache size and ensure it stays below the quota.

If the cache hit ratio is low, consider refining the key (adding more lock‑files or environment variables) or adjusting the paths list to include only the directories that truly benefit from reuse.

Actionable closing

Start by adding a cache definition to one of your most frequently changed services. Measure job times before and after, watch the cache upload/download messages, and keep an eye on storage usage. Once the pattern proves effective, replicate it across other services, adjusting the prefix in the key to match each service’s directory. Regularly review the cache size and prune stale keys to maintain a sustainable speed‑up for your monorepo pipelines.