Cutting GitLab Pipeline Feedback Loops with the `needs` Keyword

Problem: Stages Turn Good Work into Idle Time

When a team uses the classic stage‑based model in .gitlab-ci.yml, every job in a stage waits for every other job in the same stage to finish. In a typical build‑test‑deploy flow this means:

• The build stage runs for 10 min.
• Only after that does the test stage start, even if some tests don’t need the entire build output.
• Finally, the deploy stage waits for all tests to finish.

In practice, this serial chain can make a pipeline take 30 min+ while developers are waiting for feedback that could have been produced 10 min earlier.

The `needs` Keyword: A Directed Acyclic Graph (DAG) Instead of a Flat Stage

The needs keyword lets a job declare *exact* upstream jobs it depends on. The runner then builds a directed acyclic graph (DAG) of jobs, starting any job as soon as its listed dependencies are successful and its artifacts are available.

Key benefits:

• Jobs run in parallel when possible.
• Only the jobs that truly require an artifact block a downstream job.
• The pipeline wall‑clock time shrinks while maintaining explicit dependency safety.

Designing a Minimal‑Dependency Pipeline

To keep the graph readable, follow these guidelines:

1. Identify true dependencies. A job should list only upstream jobs whose artifacts or environment it needs.
2. Pass artifacts explicitly. Use artifacts:paths and needs:artifacts:true to make the artifact available to the dependent job.
3. Limit the depth of needs. Avoid chaining too many jobs with needs; deep chains can be hard to debug.
4. Document the graph. Add a comment block or a separate diagram to explain the flow for new contributors.

Concrete Example: Build, Parallel Tests, Deploy

Below is a minimal .gitlab-ci.yml that illustrates the difference between a stage‑based pipeline and a needs‑based DAG. Replace <PLACEHOLDER> with your project specifics.

# Stage‑based pipeline (slow)
stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: ./build.sh
  artifacts:
    paths:
      - dist/

unit_test:
  stage: test
  script: ./unit_test.sh
  needs: [build]

integration_test:
  stage: test
  script: ./integration_test.sh
  needs: [build]

deploy:
  stage: deploy
  script: ./deploy.sh
  needs: [unit_test, integration_test]

Because unit_test and integration_test share the same stage, they run sequentially after build finishes, even though they could run in parallel. The deploy stage waits for both to finish.

Now the needs‑based DAG:

# Needs‑based DAG (fast)
build:
  stage: build
  script: ./build.sh
  artifacts:
    paths:
      - dist/

unit_test:
  stage: test
  script: ./unit_test.sh
  needs:
    - job: build
      artifacts: true

integration_test:
  stage: test
  script: ./integration_test.sh
  needs:
    - job: build
      artifacts: true

deploy:
  stage: deploy
  script: ./deploy.sh
  needs:
    - job: unit_test
    - job: integration_test

In this configuration:

• The build job starts immediately.
• As soon as build finishes, both test jobs start concurrently.
• The deploy job starts only after both tests succeed.

Assuming build takes 10 min, each test takes 5 min, and deploy takes 2 min, the stage‑based pipeline runs for ~30 min. The needs‑based DAG runs for ~17 min—10 min for build, 5 min for parallel tests, 2 min for deploy.

Trade‑Offs and Limitations

• Graph Complexity. A large number of needs can produce a tangled DAG that is hard to read. Keep the graph shallow and document it.
• Implicit Stage Safeguards. Stages enforce that all jobs in a stage finish before the next stage starts. Using needs bypasses this safety net; missing a dependency can cause a downstream job to start before required artifacts exist, leading to flaky failures.
• Artifact Handling. needs:artifacts:true must be set for each upstream job whose artifacts you want to consume. Forgetting this flag will cause the downstream job to start without the needed files.
• Version Sensitivity. Features like needs:pipeline or needs:project are only available in GitLab 14.0+. Verify your instance’s version before using them.

Practical Checklist Before Switching to Needs

1. Run the pipeline in preview mode (commit to a branch) and inspect the Pipeline graph view to confirm the intended parallelism.
2. Compare the total duration of the stage‑based and needs‑based runs for the same commit.
3. Verify that every downstream job lists only the upstream jobs it truly requires.
4. Add a # Needs‑DAG diagram comment block in .gitlab-ci.yml for newcomers.
5. Set allow_failure: true only on jobs that are genuinely optional; otherwise consider using when: on_success to enforce safety.

Actionable Steps for Your Team

• Start small. Refactor one feature branch pipeline to use needs and measure the wall‑clock time.
• Educate. Create a short internal wiki page that explains the DAG concept and lists the most common needs patterns (e.g., build → test, test → deploy).
• Automate checks. Add a CI job that runs gitlab-ci-lint and ensures no job declares a needs on a non‑existent job.
• Monitor. Use the Pipeline analytics dashboard to track average pipeline duration before and after the change.

By moving from a flat stage model to a fine‑grained DAG with needs, teams can dramatically reduce idle time while keeping explicit, safe dependencies. The trade‑offs—complexity and the need for careful artifact handling—are manageable with a disciplined approach and clear documentation.