Making Jenkins Pipelines Resilient with the retry Directive

The Flaky Test Problem

Every CI/CD engineer knows the frustration: a pipeline fails because a third-party API timed out, a database connection dropped, or a browser test hit a transient race condition. The code is fine. The infrastructure is fine. But the build is red, and someone has to click "Rebuild" manually.

Jenkins Declarative Pipeline's retry directive solves this by automatically re-attempting failed stages or steps. Since Jenkins LTS 2.107, you can wrap flaky operations with retry(N) where N is the maximum retry count. The directive handles the backoff logic, respects quiet periods, and stops cleanly when the build is aborted.

Where retry Fits in a Declarative Pipeline

The retry block can wrap an entire stage, a single step, or a group of steps inside a script block. It only triggers on failure or unstable results—steps that complete with success or are marked always (like cleanup in post) are not retried.

Version note: This behavior is stable from LTS 2.107 onward. Earlier scripted pipelines required manual try/catch loops with sleep and currentBuild.result checks.

Worked Example: Flaky Integration Test

Consider a stage that calls an external payment sandbox. The sandbox occasionally returns 503 for a few seconds during its own deployments.

pipeline {
  agent any
  stages {
    stage('Integration Test') {
      steps {
        retry(3) {
          sh '''
            #!/bin/bash
            set -e
            ./gradlew integrationTest
          '''
        }
      }
    }
  }
  post {
    always {
      archiveArtifacts artifacts: '**/build/reports/**', fingerprint: true
    }
  }
}

Here, retry(3) wraps the sh step. If the Gradle task exits non-zero, Jenkins re-runs the entire shell block up to two more times (three attempts total). The post.always block runs after the final attempt regardless of outcome.

Where to run this: Paste into a Jenkinsfile at your repository root. Requires a Jenkins agent with Gradle installed. No special permissions beyond standard build execution.

Expected check: Open the build log. You'll see "Retry attempt 1 of 3" (then 2, then 3) prefixed to the stage output on each retry. The build status stays IN_PROGRESS until all attempts exhaust or one succeeds.

Risk: If the flakiness persists beyond three attempts, the build fails normally—no silent masking.

Combining retry with when Conditions

You can gate retries so they only apply to specific failure modes. For example, only retry on network errors, not on compilation failures:

stage('Deploy to Staging') {
  when {
    expression { return env.BRANCH_NAME == 'main' }
  }
  steps {
    script {
      retry(2) {
        sh 'kubectl apply -f k8s/staging/'
      }
    }
  }
}

The when clause controls whether the stage runs at all. The retry inside script applies only to the kubectl command. If kubectl fails with a transient API server error, it retries. If the YAML is invalid, it fails fast on the first attempt—retry doesn't change the exit code semantics.

Trade-offs and Limitations

• Queue starvation: Setting retry(10) on a long-running stage can tie up executors for hours under persistent failure. Keep counts low (2–3) and pair with timeout.
• Nested retries: A retry inside a parallel branch, or nested retry blocks, can produce surprising backoff timing. Test explicitly—Jenkins does not apply exponential backoff by default.
• No built-in delay: The directive retries immediately. If your flakiness needs a cooldown (e.g., waiting for a rate limit to reset), add sleep 30 inside the block or use a wrapper script.
• Manual retry button: The Jenkins UI still shows a "Retry" button on failed builds. That re-runs the entire pipeline, not just the failed stage. retry is finer-grained.

Verifying It Works

1. Create a test pipeline with a step that fails randomly: sh 'exit $(( RANDOM % 2 ))'.
2. Wrap it in retry(3) and run the build several times.
3. Confirm the build log shows retry attempt lines and eventually passes (or fails after three tries).
4. Compare behavior on your LTS version (e.g., 2.319 vs 2.138) if you maintain multiple controllers.

This takes five minutes and proves the directive behaves as documented before you rely on it for production workloads.

Closing Takeaway

The retry directive is a small, focused tool. It doesn't replace proper error handling, idempotent deployments, or fixing the root cause of flakiness. But it buys you time to fix those things without babysitting the build queue. Start with retry(2) on your noisiest stage, measure the reduction in manual rebuilds, and adjust from there.