Speed Up CircleCI Builds with Docker Layer Caching

Problem: Rebuilding Docker images every run wastes time

When a CI pipeline builds a Docker image from scratch on each workflow run, unchanged layers—such as the base OS or package installations—are recompiled repeatedly. This adds minutes to every build, slows feedback loops, and increases compute costs, especially for microservice repositories where the image changes only a small portion of the code.

Thesis: Turn on Docker Layer Caching to keep unchanged layers

CircleCI’s Docker Layer Caching (DLC) feature stores each layer of a built image as a separate artifact. On subsequent jobs, CircleCI restores only the layers that have not changed and pushes back any new or modified layers. When Dockerfiles are ordered with stable content early and frequently changing content later, DLC can reduce build times by 30‑70 % without altering your application code.

How Docker Layer Caching works

Each instruction in a Dockerfile creates a layer. If a layer’s contents remain identical to a previously cached version, CircleCI can reuse that layer instead of re‑executing the instruction. The feature works at the executor level: when docker_layer_caching: true is set under a machine or docker executor, CircleCI automatically:

1. Attempts to restore the layer cache at the start of the job.
2. Runs your build steps, creating or updating layers as needed.
3. Persists the updated cache after the job finishes.

If the executor type or plan does not support DLC, CircleCI silently falls back to normal layer execution—no error is shown, which makes verification essential.

Enabling DLC in your .circleci/config.yml

Below is a minimal configuration that enables DLC on the machine executor. Replace placeholders with your own values.

version: 2.1
jobs:
  build:
    machine:
      true
      docker_layer_caching: true
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Build Docker image
          command: |
            docker build -t myapp:$CIRCLE_SHA1 .
      - persist_to_workspace:
          root: .
          paths:
            - .
workflows:
  version: 2
  build_and_test:
    jobs:
      - build:
          filters:
            branches:
              only: main

Key points:

• The docker_layer_caching key must be placed directly under the executor (machine or docker).
• Your organization’s plan must be Performance tier or higher; DLC is not available on the free tier.
<
• If you use the docker executor, the same key lives under docker instead of machine.

Worked example: Comparing build times

Imagine a simple Node.js service with a Dockerfile that installs OS packages, copies package.json, runs npm ci, then copies application source.

# Dockerfile
FROM node:20-slim
RUN apt-get update && apt-get install -y \
    libssl1.1 \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json .
RUN npm ci
COPY . .
CMD ["node", "index.js"]

Because the OS‑package layer (RUN apt-get …) and the npm ci layer change infrequently, placing them before the COPY . . instruction maximizes reuse. With DLC enabled, a workflow run that only modifies src/index.js will:

1. Restore the cached OS‑package and npm ci layers.
2. Re‑run only the final COPY and any subsequent layers.
3. Save the updated top layer back to the cache.

You can observe the effect by comparing two workflow runs—one with DLC enabled and one with it disabled (by commenting out the key). The UI will show:

• Setup step: “Restoring Docker layer cache” (with a timestamp) when DLC is active.
• Persistence step: “Saving Docker layer cache” after the build.

If you do not see those messages, DLC is either not enabled or not available on your plan.

Trade‑offs and limitations

DLC is powerful but not a universal speed‑up. Consider the following:

• Plan requirement: Only Performance, Scale, or higher tiers include DLC. On lower tiers the setting is ignored silently, so you must verify your plan.
• Cache invalidation: Changing any instruction that creates an early layer (e.g., updating the base OS version) invalidates the cache for that layer and all subsequent layers, potentially removing the benefit.
• Storage cost: Each cached layer is stored as an artifact; very large numbers of layers can increase storage usage.
• Executor specificity: The cache is tied to the executor type; a job using the docker executor cannot reuse a cache built by a machine executor job.

Practical verification steps

1. UI check: Open a job’s page in CircleCI. Look for the setup step labeled “Restoring Docker layer cache” and the persistence step “Saving Docker layer cache”. Their presence confirms DLC is active.
2. API check: Use the CircleCI API to list artifacts for a job:

   curl -H "Circle-Token: $CIRCLE_TOKEN" \
     https://circleci.com/api/v2/project/gh/<ORG>/<PROJECT>/pipeline/<PIPELINE_ID>/workflow/<WORKFLOW_ID>/job/<JOB_ID>/artifacts
   

   If DLC is enabled, you will see tarball files named something like layer-0.tar.gz, layer-1.tar.gz, etc.
3. Duration comparison: Add a simple step that records the epoch time before and after the build, then compute the difference. Run the workflow with DLC toggled on and off (by editing the config) and compare the average durations over several runs.

Risk: If you accidentally enable DLC on an unsupported plan, you will see no error but also no performance gain. Always verify via the UI or API as described.

Actionable closing

To start benefiting from Docker Layer Caching today:

1. Confirm your CircleCI plan includes DLC (Performance tier or higher).
2. Add docker_layer_caching: true under the executor in your .circleci/config.yml.
3. Order your Dockerfile so that layers unlikely to change (base OS, package managers) appear early.
4. Push the change and watch the job UI for the restore/save messages.
5. Optionally, use the API or timing steps to quantify the improvement.

If you notice builds are not faster, re‑check the plan, ensure the key is correctly indented, and verify that no early‑layer changes are invalidating the cache. Adjust your Dockerfile or split heavy‑changing steps into separate stages as needed.