Dependency Caching in CircleCI: Key Design, Fallbacks, and Verification

Slow dependency installs are usually the biggest fixed cost in a CI pipeline. CircleCI's save_cache and restore_cache steps can cut that cost to seconds, but only if the cache keys are designed correctly — a bad key either never hits or, worse, restores stale dependencies silently. This guide sets up lockfile-based caching with fallback keys, then shows how to confirm the cache is actually working.

What you're building toward

By the end, your job will restore a dependency cache keyed on your lockfile's checksum, fall back to the most recent older cache when the lockfile has changed, and save a fresh cache only when the key is new. The result: unchanged dependencies restore in seconds, and changed dependencies only install the delta instead of everything from scratch.

Prerequisites

• A CircleCI project using config version 2.1 (the examples assume version: 2.1).
• A lockfile committed to the repo — package-lock.json, yarn.lock, Gemfile.lock, poetry.lock, etc. Caching without a lockfile is unreliable because there's nothing deterministic to key on.
• Permission to edit .circleci/config.yml and trigger builds on a branch.

How CircleCI caches behave (the rules that matter)

Three behaviors drive the whole design:

• Caches are immutable. Once a key is written, it can never be overwritten. A later build with the same key simply skips the save. Updating cached content requires a new key.
• Fallback keys match by prefix. If the exact key misses, restore_cache tries the next key in the list and restores the most recent cache matching that prefix.
• Lookups are branch-aware. CircleCI looks for caches on the current branch first, then falls back to the default branch. First builds on brand-new branches typically hit the default branch's cache.

Caches also have a retention window (historically around 15 days of inactivity) and size limits that vary by plan — confirm current numbers in CircleCI's docs before depending on them.

Step 1: Add restore and save steps to the job

Edit .circleci/config.yml and wrap your install step like this (Node.js example; the pattern ports to any package manager):

jobs:
  build:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-deps-{{ checksum "package-lock.json" }}-{{ arch }}
            - v1-deps-
      - run:
          name: Install dependencies
          command: npm ci
      - save_cache:
          key: v1-deps-{{ checksum "package-lock.json" }}-{{ arch }}
          paths:
            - ~/.npm
      - run:
          name: Build
          command: npm run build

Notes on the choices here:

• {{ checksum "package-lock.json" }} makes the key change automatically whenever the lockfile changes. That's the invalidation mechanism.
• {{ arch }} separates caches between CPU architectures — important if you run jobs on both x86 and Arm executors, since native modules are architecture-specific.
• The v1- prefix is your manual escape hatch (covered in recovery below).
• The fallback key v1-deps- restores the most recent cache with that prefix when the exact key misses, so a lockfile change means "install the diff" rather than "install everything".
• The example caches ~/.npm (npm's download cache) rather than node_modules. Caching node_modules directly can break when executor images change, because native modules are compiled against a specific OS and Node version. Caching the package manager's download cache is the safer default; cache node_modules only if install time still dominates and your executor image is pinned.

Equivalents for other ecosystems: ~/.cache/pip for pip, ~/.m2 for Maven, ~/.cargo/registry for Rust, vendor/bundle for Bundler (with the lockfile checksum in the key).

Step 2: Order the steps correctly

The sequence must be restore_cache → install → save_cache. save_cache is a no-op if the key already exists, so on a cache hit the save step costs nothing. On a miss (new lockfile), the install step repopulates the directory and the save writes the new key. Don't put save_cache before the install step — you'd cache an empty or stale directory.

Step 3: Verify the cache actually hits

Push the config and run the pipeline twice on the same branch. Then check three things:

1. Second run restores. Open the restore_cache step output in the CircleCI UI for the second run. You should see a message like "Found a cache from build N at key v1-deps-...". If you see "No cache is found for key", the key changed between runs — check that the lockfile is committed and the checksum path is relative to the checkout directory.
2. Lockfile change produces a new key. Modify the lockfile (e.g., bump one dependency), push, and confirm the exact key misses but the fallback prefix key restores, and the save_cache step writes the new key afterward.
3. Time actually improved. Compare total job wall time before and after. If restore takes nearly as long as the install did, your cached directory may be too large — trim it (for npm, ~/.npm/_cacache is the useful part) or accept that caching isn't a win for this job.

Recovery: invalidating a poisoned cache

Because caches are immutable, you can't edit or delete a bad one through the config — and a corrupted cache will keep restoring forever. The standard fix is to bump the version prefix in both the restore and save keys:

- v2-deps-{{ checksum "package-lock.json" }}-{{ arch }}
- v2-deps-

The next build misses every old cache, installs fresh, and saves under the new prefix. The old v1- caches age out on their own. Bump the prefix any time you change executor images, package manager major versions, or suspect bad cache contents.

Pitfalls worth knowing upfront

• Never cache directories containing credentials or tokens. Cache contents persist across jobs and branches. Keep caches to package downloads and build artifacts only.
• Concurrent saves race. If two parallel jobs save to the same key, the first one wins and the second is silently discarded. Give parallel jobs distinct key components (e.g., include the job name) if they produce different content.
• Cache size hurts you twice. Large caches are slow to save and restore and more likely to be evicted. Measure restore time; if it's over ~30 seconds, shrink what's cached.
• Exact retention and size limits change. Verify current limits in CircleCI's documentation for your plan rather than relying on any fixed number.