Hardening Production Images with Docker Multi-Stage Builds

The Problem: Bloated Runtime Images

Standard Dockerfiles often combine the build environment and the runtime environment into a single image. This results in production containers containing compilers, build-time secrets, shell utilities, and source code that are unnecessary for execution. These artifacts increase the image size—slowing down deployment and scaling—and expand the attack surface by providing tools (like curl, git, or sh) that an attacker can use after an initial compromise.

The solution is a multi-stage build: using one or more heavy-duty images to compile the application and a separate, minimal image to host the final binary. The key takeaway is that only the final FROM instruction determines the layers present in the resulting image.

Smallest Suitable Design

For most compiled languages (Go, Rust, C++) or bundled applications (Node.js, Java), a two-stage design is the most efficient baseline. This separates the Builder (SDK/Toolchain) from the Runtime (Minimal OS/Library).

Example: Go Application Architecture

# syntax=docker/dockerfile:1.7
# Stage 1: Builder
FROM golang:1.22-alpine AS builder

# Install build-time dependencies
RUN apk add --no-cache git

WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download

COPY . .
# Build a statically linked binary to avoid glibc dependencies in the runtime
RUN CGO_ENABLED=0 GOOS=linux go build -o /app/server .

# Stage 2: Runtime
FROM gcr.io/distroless/static-debian12

# Copy only the compiled binary from the builder stage
COPY --from=builder /app/server /server

# Run as a non-privileged user (Distroless static uses UID 65532)
USER 65532:65532

ENTRYPOINT ["/server"]

Trust and Data Boundaries

Multi-stage builds allow you to establish a strict boundary between build-time secrets and runtime artifacts.

Secret Isolation

Never use ARG or ENV for sensitive data like API keys or private SSH keys, as these are persisted in the image metadata. Instead, use BuildKit secret mounts. These mounts exist only during the execution of a specific RUN command and never touch the image filesystem.

Command to run: Run this on your CI runner or local terminal with DOCKER_BUILDKIT=1 enabled.

# Use a secret mount to access a .npmrc or .netrc file
RUN --mount=type=secret,id=my_token TOKEN=$(cat /run/secrets/my_token) npm install

# Build command
docker build --secret id=my_token,src=./token.txt -t my-app .

Filesystem Boundaries

The runtime image should be treated as immutable. By using a USER other than root and deploying with a read-only root filesystem, you prevent attackers from modifying binaries or installing malicious tools if they gain shell access.

Operational Checks

To verify the effectiveness of the multi-stage design, perform the following checks on the final image:

• Image Size: Compare the final image size against a single-stage version using docker images. A Go binary in distroless/static is typically <50MB, compared to >300MB for the golang:alpine image.
• Layer Inspection: Run docker history --no-trunc <image>. Verify that no RUN apk add or COPY . . commands from the builder stage appear in the history.
• User Identity: Check the configured user with docker image inspect <image> | jq .[0].Config.User. It should return a non-root UID (e.g., 65532).
• Vulnerability Scan: Use a scanner to ensure the runtime base is clean:

  trivy image --severity HIGH,CRITICAL <image>

Failure Modes and Limitations

The most common failure in multi-stage builds is the libc mismatch. If you compile a binary in a Debian-based builder (which uses glibc) and copy it into an Alpine-based runtime (which uses musl), the binary will fail to execute with a "file not found" error, even if the file exists.

Diagnostic Decision:

• If using scratch or distroless/static: Ensure the binary is statically linked (e.g., CGO_ENABLED=0 in Go).
• If dynamic linking is required: Ensure the builder and runtime share the same OS family (e.g., both based on Alpine or both based on Debian).

Design Changes

You should move away from this minimal design if:

• Debugging Requirements: If production troubleshooting requires a shell, switch from distroless to alpine or use ephemeral debug containers (e.g., kubectl debug).
• System Dependencies: If the app requires shared system libraries (like libssl or libpq), you must transition from scratch to a base image that provides those libraries.

Rollback

Since this operation changes the Dockerfile and the resulting image, rollback involves reverting the Dockerfile to the previous single-stage version and re-tagging the previous stable image version in your registry.