Architecting GitLab Runner Isolation for Shared CI/CD Environments

The Isolation Problem in Shared Runners

When multiple teams share a single GitLab Runner instance, the primary risk is cross-job contamination. If a previous build leaves a process running or a sensitive file in a shared directory, subsequent jobs may inherit that state, leading to non-deterministic build failures or security leaks. The goal is to ensure that every job starts from a known, clean state and cannot access the underlying host or other concurrent jobs.

Minimum Viable Design for Isolation

To achieve reliable isolation without over-engineering, the smallest suitable design is a GitLab Runner using the Docker Executor. In this configuration, the Runner process acts as a manager that spins up a fresh container for every single job. Once the job completes, the container is destroyed.

The architecture consists of three components:

• GitLab Instance: The orchestrator that queues jobs and provides the .gitlab-ci.yml instructions.
• Runner Manager: A lightweight Go binary installed on a VM that polls the instance via HTTPS.
• Docker Executor: The mechanism that pulls a specific image (e.g., alpine:latest) to execute the script.

Trust and Data Boundaries

Establishing clear boundaries prevents a compromised build script from taking over your infrastructure.

The Orchestrator-Executor Boundary

The Runner does not allow the GitLab server to push commands directly to the host. Instead, the Runner polls the server. This means the Runner only needs outbound HTTPS access to the GitLab instance, keeping the Runner host hidden from the public internet.

The Container-Host Boundary

By default, the Docker executor isolates the job's filesystem. However, a critical trust boundary is breached if privileged = true is set in the config.toml. Privileged mode allows the container to access host devices and potentially escape to the host OS. For shared environments, privileged mode should be disabled unless Docker-in-Docker (DinD) is strictly required for building images.

Operational Configuration Example

Below is a hardened config.toml configuration for a shared runner. This should be edited on the machine where the GitLab Runner is installed (typically as root or a user with sudo privileges).

# /etc/gitlab-runner/config.toml

[[runners]]
  name = "shared-docker-runner"
  url = "https://gitlab.example.com/"
  token = "YOUR_REGISTRATION_TOKEN"
  executor = "docker"
  
  [runners.docker]
    # Use a lightweight, secure base image
    image = "alpine:latest"
    
    # Disable privileged mode to prevent host escape
    privileged = false
    
    # Prevent containers from using the host network
    network_mode = "bridge"
    
    # Limit resources to prevent one job from crashing the host
    memory = "2g"
    cpus = "1"
    
    # Ensure a clean environment by disabling volume mounts to host paths
    volumes = ["/cache"]

Failure Modes and Recovery

Failure Mode	Impact	Detection/Mitigation
Network Partition	Jobs stay in "pending" state; Runner cannot poll.	Monitor Runner heartbeat in GitLab Admin UI.
Disk Saturation	Docker images/layers fill /var/lib/docker, causing all jobs to fail.	Implement a cron job running docker system prune -af.
Zombie Processes	Orphaned containers consume RAM/CPU.	Set timeout values in config.toml to kill hung jobs.

Verification and Testing

To verify that isolation is working, create a test job in your .gitlab-ci.yml that attempts to access the host's root directory:

test-isolation:
  script:
    - ls /root
    - hostname

Expected Result: The ls /root command should fail with "Permission denied" or show an empty directory, as it is accessing the container's root, not the host's root. If you see host-level configuration files, your isolation is compromised.

Conditions for Redesign

The Docker executor is sufficient for most tasks, but you must move to a Kubernetes Executor if:

• Horizontal Scaling: You need to scale runners dynamically based on queue depth rather than managing static VMs.
• Strict Resource Quotas: You require native Kubernetes Namespace quotas to prevent "noisy neighbor" effects across different departments.
• Complex Networking: Your builds require integration with a service mesh or complex internal DNS that Docker Bridge cannot provide.

Rollback Note: If changing the config.toml causes the runner to fail to start, revert the file to the previous version and restart the service using systemctl restart gitlab-runner.