Resolving ShotGrid API Timeouts and Pagination Performance Degradation

The Problem: Gateway Timeouts During Large Data Retrievals

When automating pipeline tools or generating reports in ShotGrid, scripts often fail with 504 Gateway Timeout errors or experience severe latency as the dataset grows. This usually occurs because the API is being asked to process too many records in a single request or is struggling with "deep paging"—the process of skipping thousands of records to reach a specific page.

The key takeaway is that ShotGrid API performance is not linear. Requesting 1,000 records in one call is significantly more prone to failure than requesting 10 batches of 100, and requesting the 10,000th record is slower than requesting the 1st.

Diagnostic Matrix: Identifying the Bottleneck

Use this table to match your observed symptom to the likely technical cause.

Symptom	Likely Cause	Diagnostic Indicator
HTTP 504 Gateway Timeout	Payload too large or complex filters	Occurs during initial request or very large limit values.
HTTP 429 Too Many Requests	API Rate Limiting	Check X-RateLimit headers in the response.
Increasing latency per page	Deep Paging (High Offset)	Request time increases as the offset value grows.
Slow response despite small record count	Over-fetching fields	Requesting all fields (default) instead of a specific subset.

Step-by-Step Resolution Path

1. Audit the Field Selection

By default, the API may return more metadata than necessary. Reducing the fields parameter minimizes the server-side processing time and the network payload size.

• Action: Explicitly define only the fields required for your task.
• Check: Compare the response time of a request for ['code'] versus a request with no fields specified.

2. Implement Strict Limit Constraints

While the API has a default limit, manually setting a conservative limit prevents the server from attempting to allocate excessive memory for a single response.

• Action: Set your limit to 100 or a maximum of 500.
• Risk: Setting a limit above 500 increases the risk of timeouts on unstable network connections.

3. Optimize Pagination Logic

If you are using a loop with offset to retrieve thousands of records, you will eventually hit a performance wall. The database must scan all preceding records before returning the requested slice.

• Action: Instead of incrementing offset, use a filter based on a sorted unique ID (e.g., id > last_seen_id).
• Verification: Run a request for the first 100 records and compare the time to a request for records 10,000 to 10,100. If the latter is significantly slower, you are suffering from deep paging.

Implementation Example: Optimized Retrieval

Run the following logic within your Python environment using the shotgun_api3 library. This example demonstrates the transition from a risky "large limit" approach to a stable "chunked" approach.

# Run this in your pipeline environment with appropriate API permissions
import shotgun_api3

sg = shotgun_api3.Shotgun("https://yourstudio.shotgrid.bpmsoftware.net", 
                         script_name="your_script", 
                         api_key="your_key")

# POOR PRACTICE: High limit, no field restriction
# data = sg.find("Asset", [["sg_status_list", "is", "Ready"]], ["*"], 1000)

# BEST PRACTICE: Small limit, specific fields, iterative retrieval
filters = [["sg_status_list", "is", "Ready"]]
fields = ["code", "sg_status_list"]
limit = 100
offset = 0
all_assets = []

while True:
    # Requesting specific fields and small chunks
    batch = sg.find("Asset", filters, fields, limit, offset)
    if not batch:
        break
    all_assets.extend(batch)
    offset += limit
    # Optional: add a small sleep here if 429 errors occur

print(f"Successfully retrieved {len(all_assets)} assets.")

Verification and Rollback

Verification: Monitor the X-RateLimit headers. If the remaining limit drops too quickly, introduce a time.sleep() between batch requests. Verify the total count of retrieved records matches the expected count from the ShotGrid web UI.

Rollback: Since this operation is a read-only find request, there is no state change to roll back. If the script fails, simply terminate the process to stop further API pressure.

When to Escalate

If you have implemented field restrictions, capped limits at 100, and avoided deep paging, but still experience 504 errors, escalate to your ShotGrid administrator or Autodesk support with the following data:

• The specific filters used (to check for unindexed custom fields).
• The exact timestamp of the timeout.
• The X-Request-ID from the response headers.