Question
Automation API preview retry limits
Lax Echo
0 reputation · 10 Jul 2022, 13:11 UTC
110.8K views1
Loading…
0 reputation · 10 Jul 2022, 13:11 UTC
The Pulumi Automation API enables programmatic stack management, allowing for the execution of previews and updates within CI/CD pipelines. When cloud provider APIs return transient errors, such as throttling or temporary unavailability, the engine implements retry logic to ensure eventual consistency.
Current behavior suggests that while exponential backoff is utilized, there is no explicit configuration within LocalWorkspaceOptions or StackOptions to define a maximum retry count or a hard timeout for these attempts during a preview operation. This can result in long-running processes that occupy pipeline runners without returning a failure state.
What is the documented method for bounding the number of retry attempts during a programmatic preview? Is there a configuration property to enforce a maximum retry limit for transient provider errors?
34275 reputation · 10 Jul 2022, 18:04 UTC
There is no documented configuration property within LocalWorkspaceOptions or StackOptions to explicitly set a maximum retry count for transient provider errors during a programmatic preview. The Pulumi Automation API acts as a wrapper around the Pulumi engine; the actual retry logic for cloud provider throttling or transient failures is handled internally by the engine and the specific provider plugins.
Because retry behavior is delegated to the provider plugin (e.g., AWS, Azure, GCP), there is no unified global setting to bound the number of attempts. These plugins typically implement their own exponential backoff strategies. Consequently, the Automation API cannot "force" a provider to stop retrying after a specific number of attempts via a high-level configuration object.
To prevent long-running processes from occupying pipeline runners, you must implement a timeout at the application level (the host language calling the Automation API) rather than the engine level. This ensures the process fails fast regardless of the provider's internal retry state.
pulumiPromise.race() wrapper or a AbortController to time out the stack.preview() call.context.WithTimeout to the Automation API methods to enforce a hard deadline.asyncio.wait_for wrapper around the preview coroutine.To verify if a process is hanging due to provider retries versus a genuine engine deadlock, check the provider logs. Most providers log "retrying request" messages when encountering 429 (Too Many Requests) or 503 (Service Unavailable) errors.
Missing Diagnostic Detail: Which specific cloud provider is experiencing the transient errors? Some providers allow timeout configurations within the provider-specific config block, though this usually applies to resource creation/deletion rather than the preview phase.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.