Troubleshooting 'Stuck' or 'Pending' Jobs in GitLab Runner

The Problem: Jobs That Won't Start

When a GitLab CI/CD pipeline triggers, jobs should transition quickly from pending to running. When a job remains stuck in a pending state, it typically means the GitLab coordinator cannot find a runner that satisfies the job's requirements or the runner is unable to communicate with the server.

Quick Diagnostic Matrix

Step 1: Verify Tag Alignment

GitLab uses tags to route jobs to specific runners (e.g., routing a macOS build to a Mac Mini). If your .gitlab-ci.yml defines tags that no active runner possesses, the job will stay pending.

• Navigate to Settings > CI/CD > Runners in your project.
• Check the tags assigned to your available runners.
• If the runner is intended to handle any job regardless of tags, ensure the "Run untagged jobs" checkbox is enabled in the runner's edit settings.

Step 2: Validate Runner Connectivity

A runner must maintain an outbound connection to the GitLab instance API to poll for new jobs. Firewalls or expired tokens often break this link.

Run the following command from the runner host to verify the API is reachable (replace gitlab.example.com with your instance URL):

# Run as a user with network access permissions
curl -I https://gitlab.example.com/api/v4/version

Expected Result: An HTTP 200 OK response. If you receive a 403 Forbidden or a timeout, check your outbound firewall rules for port 443.

Next, inspect the system logs to identify authentication errors:

# Run on the runner host with sudo/root permissions
journalctl -u gitlab-runner -n 50

Look for 401 Unauthorized errors, which indicate the registration token has expired or been revoked, requiring a re-registration of the runner.

Step 3: Audit Executor and Concurrency Settings

If the runner is online but jobs aren't processing, the bottleneck may be in the config.toml file (typically located in /etc/gitlab-runner/ or the user's home directory).

Concurrency Limits

The concurrent setting defines how many jobs the runner can execute simultaneously across all configured executors. If this is set to 1 and a long-running job is active, all other jobs will remain pending.

# Example config.toml snippet
concurrent = 4  # Increase this to allow more simultaneous jobs

# Inside the [[runners]] section of config.toml
[runners.docker]
  image = "ruby:2.7" # Default image used if none is specified in the job

# Run on the runner host with sudo permissions
sudo gitlab-runner restart