ShotGrid API: Configuring Idempotent Create Operations to Avoid Duplicate Writes

Resolution

Because the ShotGrid REST API does not support native server-side idempotency keys or atomic "upsert" operations, you must implement a Find-or-Create pattern using a custom text field as a unique identifier. This requires generating a UUID client-side, storing it in a dedicated ShotGrid field, and querying for that ID before every creation attempt.

Implementation Steps

1. Schema Setup: Create a custom text field (e.g., sg_client_id) on the target entity type. Ensure the API user has read and write permissions for this specific field.
2. Identifier Generation: Generate a version 4 UUID for the record you intend to create. This ID must persist across retry attempts for the same logical entity.
3. Pre-Flight Check: Before calling create(), execute a find() request filtering by the custom field and the UUID.

   # Conceptual Python logic
   existing = sg.find("Asset", [["sg_client_id", "is", "uuid-1234-5678"]])
   if not existing:
       sg.create("Asset", {"sg_client_id": "uuid-1234-5678", "code": "Asset_01"})
   

4. Retry Logic: If a network timeout occurs, the retry loop must start again at the Pre-Flight Check. If the previous attempt actually succeeded despite the timeout, the find() call will return the existing record, and the duplicate create() will be skipped.

Technical Considerations and Limitations

It is important to distinguish between confirmed API behavior and the inherent risks of this architectural pattern:

• Race Conditions: ShotGrid does not enforce unique constraints on custom fields. If two identical requests are sent simultaneously, both may pass the find() check before either create() call completes, resulting in duplicates.
• Rate Limiting: This pattern doubles the number of API calls per entity (one find, one create). In high-volume pipelines, this may accelerate the consumption of your API quota.
• Permissions: The API user must have explicit read access to the custom field; otherwise, the find() call will return an empty list even if the record exists, triggering a duplicate write.

Verification Process

To verify this implementation, perform the following test case:

• Initiate a create() request and manually terminate the network connection before the response is received.
• Execute the retry logic.
• Verify via the ShotGrid UI or a find() call that only one entity exists with the specified UUID.

Diagnostic Requirement: Are you executing these creates in a highly concurrent environment (multiple workers) or a sequential pipeline? If concurrent, the race condition risk is significant and may require an external locking mechanism (e.g., Redis) to ensure true idempotency.