Choosing Jenkins Pipeline Execution Environments: Docker, Kubernetes, or Static Agents

The Decision You're Facing

Every Jenkins pipeline needs a place to run. The choice between Docker agents, Kubernetes pods, and static agents determines isolation, startup latency, resource control, and ongoing maintenance. For Jenkins 2.400+ LTS, each option is fully supported but serves different operational realities. This guide lays out the constraints, compares the three approaches, and shows how to validate your selection with a test pipeline.

Constraints That Shape the Choice

• Isolation needs: Do stages require separate toolchains, conflicting dependencies, or security boundaries?
• Scale pattern: Steady baseline load, bursty spikes, or occasional hardware-specific builds?
• Infrastructure on hand: Existing Docker hosts, a managed Kubernetes cluster, or a fleet of VMs/bare metal?
• Team expertise: Comfort with container runtimes, Kubernetes RBAC, or traditional OS management?
• Latency tolerance: Can stages absorb 5–30 seconds of startup overhead, or is near-zero latency critical?

Quick Comparison

Trade-offs in Detail

Docker Agents

Use the docker-workflow plugin to run each stage in a fresh container. Example declarative syntax:

pipeline { agent none stages { stage('Build') { agent { docker 'maven:3.9-eclipse-temurin-21' } steps { sh 'mvn verify' } } stage('Test') { agent { docker { image 'node:20' args '-u root' } } steps { sh 'npm ci && npm test' } } } }

Each agent { docker ... } block pulls the image (if missing) and starts a container. The workspace is ephemeral; use stash/unstash or external artifact storage to pass files between stages. Risk: Mounting the host Docker socket (-v /var/run/docker.sock:/var/run/docker.sock) for nested builds (DinD) weakens isolation. Prefer Kaniko or BuildKit rootless for image builds inside containers.

Kubernetes Agents

The kubernetes plugin schedules a pod per stage (or per pipeline). Define pod templates in the UI or inline YAML:

pipeline { agent { kubernetes { yaml ''' apiVersion: v1 kind: Pod spec: containers: - name: maven image: maven:3.9-eclipse-temurin-21 command: ["cat"] tty: true resources: requests: memory: "2Gi" cpu: "1" limits: memory: "4Gi" cpu: "2" ''' } } stages { stage('Build') { steps { container('maven') { sh 'mvn verify' } } } } }

Pods inherit cluster RBAC; the Jenkins service account needs create/get/delete on pods. Caution: Plugin versions before 1.30.x have pod-retention and log-streaming bugs. Pin to the version matching your LTS line (check plugin compatibility matrix).

Static Agents

Permanent nodes connected via SSH, JNLP, or Windows service. Configure in Manage Jenkins → Nodes → New Node. Label them (e.g., linux-build, windows-2022) and reference in pipeline:

pipeline { agent { label 'linux-build' } stages { stage('Compile') { steps { sh 'make' } } } }

No startup delay, full host access, simplest debugging. Downside: Environment drift across nodes, manual capacity planning, OS/toolchain patching on each host. On Windows, run agent.jar as a dedicated service user (not SYSTEM) to avoid file-handle leaks.

Concrete Validation: Test Pipeline

Create a multibranch pipeline (or three separate jobs) that exercises each agent type and measures stage startup. This validates connectivity, credential masking, and resource limits before committing production workloads.

Prerequisites

• Jenkins 2.400+ LTS with docker-workflow, kubernetes, and ssh-agents (or jnlp) plugins installed.
• Access to a Docker host (local or remote) with the Jenkins user in the docker group.
• A Kubernetes cluster (v1.27+) and a service account with pod create/get/delete rights in a namespace.
• One static agent node (Linux or Windows) labeled static-test.
• A credential ID test-secret of type Secret text with value validation-token-123.

Pipeline Script

pipeline { agent none options { timestamps() } stages { stage('Docker Agent') { agent { docker { image 'alpine:3.20' args '-u root' } } steps { sh 'echo "Docker stage started at $(date)"' withCredentials([string(credentialsId: 'test-secret', variable: 'SEC')]) { sh 'echo "Secret injected: $SEC"' } } } stage('Kubernetes Agent') { agent { kubernetes { yaml ''' apiVersion: v1 kind: Pod spec: serviceAccountName: jenkins containers: - name: alpine image: alpine:3.20 command: ["cat"] tty: true resources: requests: memory: "256Mi" cpu: "100m" limits: memory: "512Mi" cpu: "200m" ''' } } steps { container('alpine') { sh 'echo "K8s stage started at $(date)"' withCredentials([string(credentialsId: 'test-secret', variable: 'SEC')]) { sh 'echo "Secret injected: $SEC"' } } } } stage('Static Agent') { agent { label 'static-test' } steps { sh 'echo "Static stage started at $(date)"' withCredentials([string(credentialsId: 'test-secret', variable: 'SEC')]) { sh 'echo "Secret injected: $SEC"' } } } } }

Where to Run & Permissions

• Create this as a Pipeline job in Jenkins UI or commit to a repo scanned by a Multibranch Pipeline.
• Requires Job/Configure permission to create the job; Credentials/View to read test-secret.
• Docker stage: Jenkins controller must reach the Docker daemon (local socket or TCP with TLS).
• Kubernetes stage: Controller needs kubectl context or in-cluster config; service account jenkins must exist in the target namespace.

Expected Checks

1. Stage startup time: Open the build console; the timestamps wrapper prefixes each line with elapsed time. Note the gap between stage start and first sh output.
2. Credential masking: Verify validation-token-123 appears as **** in console output for all three stages.
3. Resource limits:
   • Docker: On the Docker host, run docker stats --no-stream during the stage; confirm CPU/memory stay within defaults (no explicit limits set above).
   • Kubernetes: Run kubectl top pod -n <namespace> during the stage; confirm usage respects requests/limits.
   • Static: SSH to the node and run top -b -n1 | head -20; observe Jenkins executor process.
4. Agent logs:
   • Docker: Jenkins system log shows docker run --rm ... alpine:3.20 cat.
   • Kubernetes: kubectl get pod -n <ns> -o yaml shows the generated pod spec.
   • Static: Node log (Manage Jenkins → Nodes → → Log) shows java -jar agent.jar connection.

Limitations & Gotchas

• Workspace transfer: Mixing agent types in one pipeline forces stash/unstash (size-limited) or external storage (S3, Artifactory). Large artifacts (>100 MB) will slow runs.
• JCasC coverage: Configuration as Code can define static agent clouds but not dynamic Docker/Kubernetes pod templates; those remain in UI or Groovy init scripts.
• Plugin drift: The docker-workflow and kubernetes plugins release independently of Jenkins LTS. Always verify compatibility at plugins.jenkins.io before upgrading Jenkins core.
• Windows static agents: Running as SYSTEM causes file-handle cleanup failures. Create a local user, grant Log on as a service, and run agent.jar via NSSM or Windows Service Wrapper.

How to Decide Today

1. If you already run Docker hosts and need per-stage toolchain isolation without cluster overhead → Docker agents.
2. If you operate a Kubernetes cluster, want elastic scale, and align with GitOps → Kubernetes agents.
3. If you have hardware-dependent builds (GPU, FPGA, proprietary SDKs), ultra-low latency needs, or a stable VM fleet → Static agents.
4. Many teams run hybrid: build/test in Docker or K8s, deploy from a static node that holds target credentials.

Run the validation pipeline above on your infrastructure. The measured startup times, credential masking, and resource enforcement will confirm whether the theoretical trade-offs match your reality.