Speeding Up Bitbucket Pipelines with Cache Definitions and Artifacts

The problem: repeated dependency installs slow down pipelines

Every time a Bitbucket Pipelines run starts, the container is fresh. If your build step runs npm install or pip install from scratch, you spend minutes downloading and compiling the same dependencies over and over. This adds latency, increases minutes used, and makes feedback loops slower for developers.

Thesis: caches keep dependencies across runs; artifacts move files between steps

Bitbucket Pipelines offers two mechanisms that address different needs:

• Caches persist directories (like ~/.cache/pip or node_modules) across separate pipeline executions for the same branch. They are uploaded at the end of a successful run and downloaded at the start of the next run.
• Artifacts pass files between steps within the same pipeline run. They are useful for handing off build outputs, test reports, or Docker images to later stages.

By combining a cache for immutable dependencies with artifacts for mutable build results, you can cut install time dramatically while keeping each step isolated.

Worked example: caching Node.js dependencies and passing build artifacts

Assume a repository with a Node.js project that uses npm ci to install exact versions from package-lock.json. The pipeline has three stages: install, test, and build.

Step 1: declare a reusable cache in the definitions block

# bitbucket-pipelines.yml
images:
  default: node:20

definitions:
  caches:
    node: ~/.npm
    # Note: $HOME expands to /opt/atlassian/pipelines/agent/build in the default image
  # end definitions

pipelines:
  default:
    - step:
        name: Install dependencies
        caches:
          - node
        script:
          - npm ci
        artifacts:
          - node_modules/**
    - step:
        name: Run tests
        script:
          - npm test
    - step:
        name: Build application
        script:
          - npm run build
        artifacts:
          - dist/**

Explanation:

• The definitions.caches.node entry names the cache node and points to ~/.npm, where npm stores its cache and the node_modules folder when using npm ci.
• The Install dependencies step references caches: - node. On the first run, the cache is empty; after npm ci finishes, Pipelines uploads the ~/.npm directory.
• On subsequent runs for the same branch, Pipelines downloads the cache before the step starts, restoring ~/.npm and making npm ci nearly instantaneous.
• The artifacts section in the install step saves node_modules so the test step can access it without reinstalling. The build step publishes the dist folder as an artifact for later deployment or download.

Where to run and what permissions are needed

Push the updated bitbucket-pipelines.yml to the repository (requires write access). Then trigger a pipeline manually from the Pipelines UI or by pushing a commit. No special admin rights are needed beyond standard repository write permissions.

Trade‑offs and limitations to keep in mind

• Size limits: Bitbucket Cloud enforces a per‑repository cache size quota (currently around 1 GB). If the cached directory exceeds this, the upload may be silently truncated, leading to missing dependencies and build failures. Monitor the pipeline log for messages like "Cache size exceeded limit" and consider pruning unused packages or splitting caches.
• Branch scope: A cache created on main is not automatically available to a feature branch until that branch runs a pipeline and creates its own cache. For short‑lived branches, expect a cold install the first time.
• Manual invalidation: Caches persist until you change the cache definition name. A common pattern is to append a checksum of the lock file, e.g., node-{{ checksum "package-lock.json" }}. If you forget to update the name after a dependency change, the stale cache will be used, potentially causing hard‑to‑debug errors.
• Secrets risk: Never store environment files containing tokens or passwords inside a cached directory. Caches can be restored in contexts with different permissions, exposing secrets to unintended steps.
• Image dependency: The cache path must exist in the Docker image used. If you switch from node:20 to an image where $HOME points elsewhere, the cache restore will fail. Keep paths absolute or use $HOME which is consistent across official Node images.

Actionable closing: add, verify, iterate

1. Add a definitions.caches block for your language’s package manager (e.g., ~/.cache/pip for Python, ~/.m2 for Maven).
2. Reference the cache in the install step and watch the pipeline log for "Restoring cache" and "Uploading cache" lines.
3. Compare the duration of the install step between the first run (cold) and the second run (warm). A noticeable reduction confirms the cache works.
4. To test invalidation, change the cache name (e.g., add a version suffix) and verify the next rebuild performs a fresh install.
5. Periodically check the repository’s Pipelines usage page for cache size warnings and adjust what you cache if you approach the limit.

By treating caches as a shared dependency store and artifacts as the handoff mechanism between steps, you can make Bitbucket Pipelines faster and more predictable without sacrificing isolation.