Jenkins Pipeline and External API Integration: Idempotency during retry(N) execution

Question

The retry(N) step in Jenkins Pipeline is designed to re-execute a block of code upon failure. However, this mechanism does not track side effects or state changes that occurred during previous attempts within the same block.

When integrating Jenkins with external systems—such as REST APIs or databases—a failure occurring after a write operation but before the step completes results in the write being issued again on the next retry. This creates a risk of duplicate records or inconsistent state if the external system does not natively support idempotency.

Currently, Jenkins does not provide a built-in way to generate idempotency keys or track successful external writes across retry attempts, leaving the deduplication logic entirely to the pipeline script or the target system.

Is there a documented method to maintain a persistent state for a specific retry block to skip already-completed write operations?
  Should this deduplication logic be handled via a dedicated plugin, or is the current design intended to rely exclusively on external system idempotency?

Tasadduq Burney · Answer

Direct answer
No. Jenkins Pipeline's retry(N) step does not provide documented, attempt-scoped persistent state that records which external writes already succeeded. When an exception occurs after a write but before the block returns, the next attempt re-executes the entire block, including the write. Jenkins also does not generate idempotency keys automatically.
Confirmed facts and likely behavior
Confirmed: retry(N) re-runs the whole block after any exception. Jenkins Pipeline does not expose durable, retry-block-scoped storage for tracking completed side effects.
Likely: A variable declared outside retry can survive in memory for the rest of one build, but it is not transactional and may be lost on CPS serialization, agent loss, or controller restart. A shared-library helper can create a UUID before the retry and include it in every request, but that only helps if the receiving system recognizes and deduplicates the key.
Steps for this case

Generate a stable idempotency key before calling retry, for example with a UUID.
Send that same key with every attempt, typically in an Idempotency-Key header or request body field.
Confirm the external API or database deduplicates on that key, or supports conditional or unique writes. If it does not, do not retry the block after a side effect has occurred.
Keep idempotent calls such as reads inside retry. Place state-changing writes before retry or behind an external deduplication layer.

Verification
Run a test job with retry(2) against a mock API that logs every request. Fail the step after the mock records the write but before it returns success. Confirm the mock receives a second call unless your pipeline sends a stable idempotency key and the mock deduplicates it.
Missing diagnostic detail
Does the external API accept an idempotency key, or does it support conditional inserts or unique constraints? This determines whether you can safely retry with a key or must avoid retrying after writes entirely.
Assumptions and uncertainty
This answer assumes standard Jenkins Pipeline behavior with the Pipeline plugin installed. Exact step semantics can vary by plugin version and Groovy sandbox approvals. Storing keys in Jenkins may expose authorization material; treat them as sensitive and restrict job access. No dedicated plugin is required, and no plugin can make an arbitrary third-party API idempotent by itself.

Jenkins Pipeline and External API Integration: Idempotency during retry(N) execution

1 answer

Direct answer

Confirmed facts and likely behavior

Steps for this case

Verification

Missing diagnostic detail

Assumptions and uncertainty

Your perspective belongs here.

0 question comments