vCenter Server API and External Automation Client: safe retry for mutating writes without duplicates

Answer the question directly

To decide whether a retry should reuse an existing task, match the new request against recent tasks on the target object using these reliable criteria:

1. Target Managed Object Reference (MoRef) – the object the operation acts on (VM, Folder, Datastore, etc.).
2. Operation name – the method invoked (e.g., CreateVM_Task, ReconfigureVM_Task).
3. Start time window – the task’s info.startTime must fall within a bounded look‑back period (see polling window guidance below).
4. Input arguments (when exposed) – for many tasks the info.descriptionId or info.taskInProgress fields contain enough detail to distinguish calls; if the API exposes the argument values (e.g., VM name, disk size) compare them for equality.
5. Task state – accept only tasks in success or running states; treat error as a failed attempt.

Steps for a safe retry

1. Generate a client‑side UUID (request ID) before the first call and store it.
2. Issue the mutating operation (e.g., CreateVM_Task) and record the returned Task MoRef.
3. Poll the task’s info.state until it reaches success, error, or a timeout occurs.
4. If a transient transport failure occurs before a final state is observed:
• Query recentTasks on the target MoRef (or on the root Folder for VM‑creation tasks) filtering by:
• Operation name matching the original call.
• Start time ≥ now − pollingWindow.
• Optional: argument equality if the task description exposes them.
• If a task is found and its state is success, treat the retry as successful and return the stored result.
• If a task is found and its state is running, continue polling that task until it finishes or exceeds the polling window.
• If no matching task is found within the window, or the matching task ends in error, it is safe to retry the operation (using the same client request ID).
5. Bound the polling window for an “orphaned” task: use a value of 2 × expectedMaximumDuration for the specific operation (e.g., 10 min for a typical VM create, 2 min for a disk create). This window is long enough to capture a task that survived a brief network partition but short enough to avoid waiting indefinitely on a truly failed call.

Confirmed facts vs. likely explanation

• Confirmed fact: vCenter does not guarantee exactly‑once semantics for mutating Task‑based APIs across network partitions or failover; duplicates can occur if a retry is issued without checking existing tasks.
• Confirmed fact: Starting with vSphere 7.0 U3, certain REST endpoints honor the X‑VMWARE‑VCENTER‑REQUEST‑ID header for idempotency, but this header is not universally implemented for all SOAP/Task‑based calls.
• Likely explanation: When the request‑ID header is not honored, client‑side deduplication using the criteria above provides the best practical guarantee of no duplicate side‑effects.
• Likely explanation: The polling window should be derived from observed operation latency; if you lack that data, start with a conservative estimate (e.g., 5 min for most VM‑level mutating calls) and adjust based on monitoring.

Missing diagnostic detail

To tighten the polling window recommendation, please provide the typical observed duration (or SLA) for the specific mutating operation you are retrying (e.g., average time to complete CreateVM_Task in your environment). This will let us calculate a precise 2 × duration bound.