Investigate how GitLab's configured API rate‑limit ceiling influences overall CI/CD pipeline throughput when max concurrent pipelines is adjusted. The default user‑level limit of 1000 requests per minute, configurable per instance via the rate_limit setting, interacts with pipeline concurrency controls under admin settings. When pipeline execution generates API calls that exceed this threshold, HTTP 429 responses with Retry‑After headers may stall job startups, and the extent to which pipeline concurrency settings amplify or mitigate this effect remains undocumented for mixed‑user runner environments. Key uncertainties include whether the rate‑limit applies per‑user, per‑project, or per‑instance during concurrent pipeline runs, and how Retry‑After timing impacts queuing behavior when multiple pipelines compete for API slots. Differences between GitLab.com and self‑hosted instances further complicate threshold predictions, as do variations in runner registration scope and token scope. What measurable threshold exists where increasing max concurrent pipelines ceases to improve throughput under the default 1000 requests per minute user limit? Does the Retry‑After header behavior differ when the limit is exceeded during pipeline startup versus mid‑execution job steps? How do per-instance rate‑limit overrides alter pipeline queuing when multiple projects share a single runner under a common user context?

GitLab Rate-Limit vs Pipeline Concurrency

1 answer

34775 reputation · 25 Feb 2020, 14:49 UTC

Answer first

No universal measurable threshold exists for max concurrent pipelines under the default 1000 requests per minute user limit. Throughput stops improving when the aggregate API call rate of active jobs sharing the same authenticated user approaches the per-user ceiling, and that point depends on job chattiness, runner count, and instance tuning. Retry-After behavior is the same mechanism at startup and mid-execution; the visible symptom differs. Per-instance rate-limit overrides move the ceiling for all users sharing a token context, which matters most when multiple projects run on a runner authenticated under one user.

Confirmed facts

GitLab’s authenticated API rate limit is applied per user, and unauthenticated requests are per IP. It is not per project or per pipeline.
On GitLab.com the authenticated limit is on the order of 1000 requests per minute per user. Self-hosted instances can adjust or disable limits via admin application settings.
When the limit is exceeded GitLab returns HTTP 429 with a Retry-After header. Well-behaved clients, including GitLab Runner’s API polling, back off. Jobs are delayed, not failed outright.

Likely explanation

Runner-to-GitLab traffic consumes API calls per active job: job pickup, status updates, trace uploads, artifact handling. Raising max concurrent pipelines increases concurrent jobs, which raises API pressure roughly linearly. With a shared user context, concurrent pipelines compete for one request budget. Once the budget is saturated, added concurrency converts into 429-induced waiting rather than completed work, so pipeline duration and pending time rise while completed jobs per minute plateau.

Retry-After semantics do not change between pipeline startup and mid-execution. At startup throttling shows as jobs staying pending longer. Mid-execution throttling stretches trace streaming and artifact uploads, increasing job wall time without failing the job.

A single shared runner serving multiple projects under one user token concentrates all API consumption into one bucket. In that topology per-instance limit overrides are the primary lever for relieving queuing; pipeline concurrency controls alone will not help once the user bucket is saturated.

Steps needed for this case

Confirm scope and value. Check admin rate-limit settings and make a throttled request to see the actual limit and response headers for your instance.
Identify the authentication context. Note which user or bot token the runner uses for API calls and whether multiple projects share it.
Measure empirically. On a test instance, ramp concurrent pipelines under one user and record 429 counts, Retry-After values from runner logs and GitLab access logs, and end-to-end pipeline duration and job pending time per concurrency level.
Find your plateau. The concurrency level where pending time rises and completed jobs per minute stop increasing is your practical limit for that workload.

One missing diagnostic detail that changes the recommendation: which user token and runner registration scope is used for the API calls in your mixed-user runner environment. If the runner uses a single bot user for all projects, the per-user bucket is shared and raising max concurrent pipelines will hit the limit sooner. If each project uses its own token/user, the limit is effectively partitioned.

Your perspective belongs here.

Join the conversation

Rate-Limit Threshold Interaction With Pipeline Concurrency