Speeding Up CircleCI Pipelines: Deterministic Caching and Workspace Persistence

Pipelines that rebuild Docker layers on every run waste precious minutes—especially on CircleCI’s shared runners. The cache key and persist_to_workspace are the platform’s built-in tools to avoid redundant work, but they’re easy to misconfigure. Used intentionally, they can make subsequent runs noticeably faster; misused, they cause frequent cache misses or truncated artifacts.

\n

How the cache key works

\n

The cache key directive tells CircleCI to store and restore directories across pipeline runs. When the key matches—typically via a checksum of input files—CircleCI reuses a previously saved directory, skipping Docker layer rebuilds for that content. If the key changes (because a file shifted or a checksum command became non-deterministic), the cache misses and the pipeline starts from scratch.

\n

Key design patterns

\n

\n
• Pin the key to a stable artifact: key: v1-{{ checksum 'package-lock.json' }}
\n
• Include only directories that truly repeat across runs: paths: - ~/.cache - node_modules
\n
• Avoid checksumming generated outputs or volatile environment values; those produce new keys every time.
\n

\n

Pairing cache with persist_to_workspace

\n

Some dependencies or build outputs live outside Docker layers—compiled binaries, generated reports, or large data files. The persist_to_workspace job attribute saves directories to the pipeline workspace, and attach_workspace makes them available in downstream jobs. CircleCI caps the workspace at 150 MB per pipeline; exceeding it causes the run to fail or silently truncate.

\n

Practical guardrail

\n

Only persist what’s essential. If a directory exceeds a few hundred megabytes, consider uploading it as a CircleCI artifact instead (store_artifacts), or split the persistence across multiple jobs.

\n

Remote caching via Orbs

\n

For teams running on shared runners, CircleCI Orbs can expose remote caching layers. An orb like circleci/cache can store keys in a persistent backend, so cache hits persist across different runner instances and even across team members’ pipelines. Setup typically adds an orb reference and ensures the same deterministic key is used across jobs.

\n

Worked example: a minimal caching config

\n

version: 2.1\norbs:\n  node: circleci/node@5\njobs:\n  build:\n    docker:\n      - image: cimg/node:20.10\n    steps:\n      - checkout\n      - cache-restore:\n        keys:\n          - v1-{{ checksum 'package-lock.json' }}\n      - run:\n          name: Install dependencies\n          command: |\n            npm ci\n      - save_cache:\n        key: v1-{{ checksum 'package-lock.json' }}\n        paths:\n          - ~/.npm\n      - persist_to_workspace:\n        root: /tmp/workspace\n        paths:\n          - . # adjust; respect the 150 MB limit\nworkflows:\n  main:\n    jobs:\n      - build

\n

What to check

\n

\n
• After the first run, inspect the CircleCI UI for a cache hit indicator next to the save_cache step.
\n
• On the second run of the same pipeline, the cache-restore step should report a hit, and the Install dependencies step may complete faster because Docker layers for npm ci are reused.
\n
• If the key changes (e.g., you modify package-lock.json without updating the checksum pattern), the UI will show a cache miss and the pipeline will rebuild the layers.
\n

\n

Relevant risk

\n

Using {{ checksum }} on a file that changes unpredictably—such as a timestamp‑based version or a dynamically generated manifest—will defeat the cache. Always anchor keys to files that follow a predictable update cadence.

\n

Trade‑off and limitation

\n

\n
• Deterministic keys are a manual task. A poorly designed key results in cache misses on every run, negating the benefit.
\n
• 150 MB workspace cap. Large build outputs or binary artifacts can exceed this, causing truncation failures. Use store_artifacts for items above the threshold.
\n
• Cache invalidation. CircleCI does not automatically expire stale caches; a key change is the only way to force a refresh. This means old caches can linger if keys aren’t versioned.
\n

\n

Actionable closing

\n

Start by auditing one job in your pipeline: identify a directory that’s rebuilt every run, add a cache key anchored to a checksum of its inputs, and run the pipeline twice. Check the UI for the cache hit badge. If the key design is solid, you’ll see faster subsequent runs. Gradually extend the pattern to other jobs, and remember to offload anything beyond 150 MB to artifacts. The result is a pipeline that reuses work instead of repeating it—without requiring runner upgrades or complex infrastructure.