Enable Docker BuildKit for Lean Multi‑Stage Builds

Desired Outcome

Activate Docker’s BuildKit engine to write efficient, cross‑platform multi‑stage builds that reduce image size and expose build progress and cache details.

Prerequisites

• Docker Engine 18.09+ (BuildKit is experimental in 18.09 and fully supported from 19.03 onwards).
• Root or sudo privileges to edit /etc/docker/daemon.json if you choose a daemon‑level setting.
• Basic familiarity with Dockerfile syntax.
• Optional: docker buildx plugin for advanced builder management.

Enabling BuildKit

You can enable BuildKit temporarily for a single build or permanently for all builds.

One‑Shot Activation

# Run in the terminal where you invoke docker build
export DOCKER_BUILDKIT=1
# or for a single command
DOCKER_BUILDKIT=1 docker build -t myapp:latest .

Setting the environment variable only affects the current shell session. The build will use the new builder and show BuildKit‑specific output.

Daemon‑Level Activation

Edit /etc/docker/daemon.json (create it if missing) and add a builder section:

{
  "builder": {
    "default": {
      "gc": {
        "defaultKeepDuration": "24h",
        "enabled": true
      }
    }
  }
}

After saving, restart Docker:

sudo systemctl restart docker

Verify BuildKit is active with:

docker version --format "{{.Server.Experimental}}"
# Expected output: true

Crafting a Multi‑Stage Dockerfile

Below is a minimal example that builds a Go application, targets a specific platform, and copies only the final binary into a lightweight runtime image.

# syntax=docker/dockerfile:1.4

# ---- Builder Stage -----------------------------------------------------
FROM --platform=linux/amd64 golang:1.22-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o /app/bin/myapp

# ---- Runtime Stage -----------------------------------------------------
FROM alpine:3.20
WORKDIR /app
COPY --from=builder /app/bin/myapp ./myapp
EXPOSE 8080
CMD ["./myapp"]

Key points:

• --platform=linux/amd64 forces a Linux/AMD64 image, useful for cross‑platform CI.
• All build tools live in the builder stage; the final image contains only the binary.
• BuildKit understands the --from syntax and will not copy intermediate layers into the final image.

Executing the Build

Use --progress=plain to expose BuildKit’s detailed output, useful for debugging and performance tuning.

docker build --progress=plain -t myapp:latest .

Typical output will include lines like:

[1/4] FROM golang:1.22-alpine AS builder
[2/4] WORKDIR /src
[3/4] COPY go.mod go.sum .
[4/4] RUN go mod download
[5/4] COPY . .
[6/4] RUN go build -o /app/bin/myapp
[7/4] FROM alpine:3.20
[8/4] WORKDIR /app
[9/4] COPY --from=builder /app/bin/myapp ./myapp
[10/4] EXPOSE 8080
[11/4] CMD ["./myapp"]

Notice the stage names (builder, final) and the cache hit/miss indicators that BuildKit provides.

Verification Steps

1. Inspect the image size – the final image should be significantly smaller than the builder image.

   docker image inspect myapp:latest --format '{{.Size}}'
   # Compare against the intermediate image size
   

2. Check the filesystem – ensure only the expected files are present.

   docker run --rm myapp:latest ls -l /app
   # Should list only myapp binary and any runtime assets
   

3. Optionally, use docker buildx ls to verify the active builder.

   docker buildx ls
   # Look for a builder with driver=local and default set
   

Recovery & Fallback

If BuildKit produces errors (e.g., unknown syntax or missing stages), the simplest rollback is to disable it for that build:

DOCKER_BUILDKIT=0 docker build -t myapp:latest .

Or, if you enabled it daemon‑wide, revert the daemon.json change and restart Docker. The legacy builder will process the Dockerfile but will not support the --platform flag or inline cache features.

CI Integration Tip

In a CI job, you can build and push in one step, ensuring the same image is used across environments:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --push \
  -t registry.example.com/myapp:latest \
  .

Use --load instead of --push if you only need the image locally on the build host.

Limitations & Caveats

• Older Docker Engine versions ignore DOCKER_BUILDKIT and silently fall back to the legacy builder, which does not understand the multi‑stage syntax.
• BuildKit exposes cache data on the host; if sensitive artifacts are copied into intermediate stages, consider --secret or multi‑stage cleanup.
• Verbose --progress=plain logs can leak build arguments; avoid logging them in public CI systems.

Practical Check

After building, run:

docker run --rm myapp:latest ./myapp --version

If the command exits successfully and outputs the expected version string, the build and copy steps succeeded.