Diagnosing CircleCI Cache Misses: A Step-by-Step Guide

The Problem: Cache Misses Slow Down Every Build

CircleCI’s save_cache and restore_cache steps are the primary lever for reducing dependency-install time. When they miss, every job re-downloads packages, adding minutes per workflow. The symptoms are consistent: CACHE_HIT=false in step output, repeated npm ci or pip install runs, and quota warnings that appear only after the cache has already failed to save.

Recognizable Conditions

Symptom	What You See in Logs
Cache miss on every commit	restore_cache prints CACHE_HIT=false; install step runs every time
Cache saved but never restored	save_cache succeeds, next build still reports miss
Quota exceeded warning	Warning: Cache quota exceeded in save_cache step; subsequent saves become no-ops
Stale artifacts after restore	Dependencies present but outdated; build fails with version conflicts
Cross-OS corruption	Windows/macOS jobs restore a Linux cache (or vice‑versa) and crash on native modules

Cause & Diagnostic Table

Root Cause	Diagnostic Check	Evidence in Logs
Key lacks deterministic inputs (lockfile, package-manager version, OS image)	Inspect restore_cache keys in .circleci/config.yml	Key uses only {{ .Branch }} or {{ .Environment.CIRCLE_SHA1 }} without checksum
Overly broad key causing bloat and eviction	List caches via API; look for many entries with same prefix	Hundreds of keys like v1-deps-main-; total size near 5 GB
Overly narrow key (full SHA) preventing reuse	Key includes {{ .Environment.CIRCLE_SHA1 }}	Every build creates a unique cache entry; hit rate near 0%
Cache size > 2 GB soft limit or 5 GB repo quota	Run API query (see Verification); check save_cache warnings	Warning: Cache ... is larger than 2GB or Cache quota exceeded
Fallback chain missing or misordered	Review restore_cache keys list	Only one key listed; no generic fallback like v1-deps-
Parallelism sharding without per‑shard keys	Job uses parallelism > 1; key lacks CIRCLE_NODE_INDEX	All shards restore same cache, then overwrite each other on save
Docker layer caching (DLC) confused with save_cache	Job uses setup_remote_docker; Dockerfile order changed	docker build shows no CACHED layers; save_cache unrelated
Race condition: background writes still running during save_cache	Check for background processes (e.g., npm install &) before save step	Intermittent corrupt archives; cache restores but files missing

Ordered Troubleshooting Checks

1. Validate key design. Open .circleci/config.yml. A correct Node key looks like:

   restore_cache:
     keys:
       - v1-deps-{{ checksum "package-lock.json" }}-{{ .Environment.CIRCLE_NODE_INDEX }}
       - v1-deps-{{ checksum "package-lock.json" }}
       - v1-deps-{{ .Branch }}
       - v1-deps-
   save_cache:
     key: v1-deps-{{ checksum "package-lock.json" }}-{{ .Environment.CIRCLE_NODE_INDEX }}
     paths:
       - ~/.npm

   Replace package-lock.json with pom.xml, build.gradle, requirements.txt, or go.sum for other ecosystems.
2. Measure cache size and quota. Run the API query (requires a personal API token):

   curl -H "Circle-Token: $CIRCLE_TOKEN" \
     https://circleci.com/api/v2/project/gh///caches | jq '.items[] | {key: .key, size_mb: (.size/1024/1024|floor), last_used: .last_used}'

   If total size approaches 5 GB or any entry exceeds 2 GB, split caches by service or prune old keys.
3. Confirm fallback chain works. Push a commit that changes the lockfile. The job should restore the v1-deps-{{ .Branch }} or v1-deps- key, then run the install step to update. Verify the install step executes (it must not be skipped).
4. Check parallelism interaction. If parallelism: 4 is set, ensure each shard saves a distinct cache:

   key: v1-deps-{{ checksum "package-lock.json" }}-{{ .Environment.CIRCLE_NODE_INDEX }}

   For a unified dependency cache, use the same key without NODE_INDEX and accept that only the last shard’s save wins (usually fine).
5. Separate DLC issues. If the job uses setup_remote_docker, add a debug step:

   - run: docker history  --format "{{.CreatedBy}}" | head -20

   Look for CACHED in the build output. DLC misses are fixed by ordering Dockerfile instructions from least to most changing and pinning base image digests.
6. Guard against cross‑OS corruption. Include executor type in the key:

   key: v1-deps-{{ checksum "package-lock.json" }}-{{ .Environment.CIRCLE_JOB }}-{{ .Environment.CIRCLE_NODE_INDEX }}

   CIRCLE_JOB expands to the job name (e.g., build-linux, build-macos).
7. Eliminate race conditions. Ensure all background processes finish before save_cache. Use wait or restructure steps so installation completes in the foreground.

Fixes Tied to Findings

Finding	Fix	Config Change
Key missing checksum	Add checksum "lockfile" to every key	Replace v1-{{ .Branch }} with v1-{{ checksum "package-lock.json" }}
Cache bloat from broad keys	Narrow keys to lockfile checksum; delete old keys via UI (Project Settings → Caches)	Change prefix to v2- to force fresh namespace
Quota exceeded	Split monorepo caches per service; remove unused branches	Multiple save_cache steps with distinct paths and keys
No fallback chain	Add generic keys to restore_cache.keys list	Append - v1-deps-{{ .Branch }}, - v1-deps-
Parallelism overwrites	Include CIRCLE_NODE_INDEX in key	Key suffix -{{ .Environment.CIRCLE_NODE_INDEX }}
DLC miss	Reorder Dockerfile; pin base image digest	FROM node:20@sha256:… instead of node:20
Cross‑OS restore	Add job name or OS tag to key	Suffix -{{ .Environment.CIRCLE_JOB }}
Race condition	Move save_cache after all installs; remove background operators	Ensure npm ci runs in foreground

Verification Steps

1. Add a debug step immediately after restore_cache:

   - run:
       name: Cache debug
       command: |
         echo "CACHE_HIT=$CACHE_HIT"
         ls -la ~/.npm 2>/dev/null || echo "no npm cache"
         du -sh ~/.npm 2>/dev/null || true

   Confirm CACHE_HIT=true on unchanged lockfile and non‑zero cache size.
2. Run circleci config validate locally (CircleCI CLI) to catch interpolation errors before pushing.
3. After a successful build, re‑run the API query from step 2. The target key should show a recent last_used timestamp and size under 2 GB.
4. Simulate a miss: edit a lockfile, push, and verify the fallback key restores a usable (stale) cache and the install step runs to update it.

Escalation Criteria

• Cache quota exhausted despite splitting caches and pruning keys — contact CircleCI support to request a quota increase (available on Scale plans).
• Persistent save_cache no‑ops with no warning in logs — may indicate a platform‑side storage issue; open a support ticket with build URLs.
• Corrupted caches that survive key rotation — verify no background writes; if clean, report as a potential runner/storage bug.

Limitations & Gotchas

• Cache quota is shared across all projects in an organization on some plans. A single noisy repo can starve others. Monitor via the API or UI (Plan → Cache Usage).
• Caches are not encrypted beyond platform defaults. Never store secrets, tokens, or generated credentials in cached paths.
• No programmatic invalidation API. To force a refresh, change the key prefix (e.g., v2-) or delete manually in Project Settings → Caches.
• Windows and macOS executors have different filesystem semantics (case sensitivity, symlinks). Always include executor identifier in keys.
• DLC is entirely separate from save_cache/restore_cache. DLC requires setup_remote_docker and a remote engine; its misses are diagnosed via docker history, not the cache API.

Quick Reference: Good vs. Bad Key Patterns

Pattern	Result	Example
Bad: only branch	Miss on every commit	v1-deps-{{ .Branch }}
Bad: full SHA	Never reused	v1-deps-{{ .Environment.CIRCLE_SHA1 }}
Good: lockfile checksum + shard	Deterministic, reusable, parallel-safe	v1-deps-{{ checksum "package-lock.json" }}-{{ .Environment.CIRCLE_NODE_INDEX }}
Good: fallback chain	Graceful degradation	keys: [specific, v1-deps-{{ .Branch }}, v1-deps-]