Diagnosing ShotGrid API Throttling: Rate Limits, N+1 Queries, and the Fixes That Actually Work

The recognizable condition

Your custom tools, pages, or daemons that call the ShotGrid API (now branded Autodesk Flow Production Tracking) used to be fine, but now they slow down and intermittently fail with HTTP 429 responses. Failures cluster during heavy pipeline activity — morning publishes, bulk version uploads, dailies crunch. Dig into the client logs and a pattern emerges: one API call per row in a table, or one call per linked entity, multiplied by every user action. That is self-inflicted throttling, and it is almost always fixable without changing your plan.

The underlying trap is how the API returns linked entities. A find() on Versions returns each linked Shot as a shallow dict like {\"type\": \"Shot\", \"id\": 1234, \"name\": \"010_0010\"}. If your grid needs the Shot's code, status, and Sequence, the tempting move is a follow-up find() per row. That is the classic N+1 problem, and against a per-site rate limit it produces exactly the 429 bursts you are seeing.

Cause and diagnostic table

Ordered checks

1. Enable HTTP-level logging in your client and count calls per user action. A single page render that fires 200 requests is the smoking gun.
2. Inspect one raw response payload. If linked entities come back as shallow {type, id, name} dicts and your code then fetches them, you have N+1.
3. Search the codebase for API calls inside loops — for blocks, list comprehensions, and per-row callbacks.
4. Confirm the installed client library version (for example shotgun_api3) and check its changelog or source for 429 retry and backoff behavior. Older versions may retry tightly, making throttling worse.
5. Review pagination: look for find() calls with no limit, or very large page sizes that time out.
6. For automation, check the site's event backlog or webhook delivery logs before assuming your script is broken.

Fixes tied to findings

N+1 fetches: ask for linked fields with dot notation

The API supports requesting fields through a link in a single query. Instead of fetching Versions and then each Shot, request the Shot's fields inline:

# Run anywhere with network access to your site and a valid script key.\n# Requires: pip install shotgun_api3\nimport shotgun_api3\n\nsg = shotgun_api3.Shotgun(\n    \"https://YOUR-SITE.shotgrid.autodesk.com\",\n    script_name=\"YOUR_SCRIPT\",\n    api_key=\"YOUR_API_KEY\",\n)\n\nfields = [\n    \"code\",            # Version code\n    \"entity\",          # shallow link to the Shot\n    \"entity.Shot.code\",        # Shot code, resolved server-side\n    \"entity.Shot.sg_status_list\",\n    \"entity.Shot.sg_sequence.Sequence.code\",\n]\nversions = sg.find(\"Version\", filters=[[\"project\", \"is\", PROJECT_DICT]], fields=fields, limit=50)

One call now returns everything a row needs. To verify the win, log HTTP requests while running the old and new versions of the query — you should see the request count drop from 1 + N to 1. Keep traversals shallow and targeted: very deep or broad dot-notation requests shift cost into a single slow query rather than removing it.

Many writes: combine them with batch()

If logs show bursts of individual create/update calls, use sg.batch() to send multiple operations in one request. Important caveat: batch() is not a transaction. Partial failure semantics vary by client library version, so test what a mid-batch failure returns before relying on it for multi-step writes, and re-check entity state after any reported failure.

429s: honor backoff instead of retrying tightly

Replace immediate retry loops with exponential backoff, and respect any Retry-After hint the response carries. Exact thresholds and headers vary by plan and have changed over time, so check the current API documentation rather than hardcoding numbers.

Repeated reference lookups: cache with a TTL

Projects, users, and status lists change rarely. Cache them in memory or on disk with a time-to-live (minutes to hours) instead of re-fetching per action.

Polling loops: go event-driven

Polling scales poorly because call volume grows with site activity regardless of whether anything changed. Two supported paths:

• Webhooks: configure the site to push events to your endpoint. Validate by triggering a test event and confirming delivery and latency in the site's webhook logs.
• Event daemon: consume EventLogEntry records with the open-source shotgun-events daemon. Note this is community-supported, not an official Autodesk support channel — factor that into your escalation plan.

The engineering decision underneath

Treat the ShotGrid API as shared, rate-limited infrastructure, not a private database. Prefer fewer, wider queries over many narrow ones, and design integrations event-driven rather than poll-based. Polling is the single most common source of self-inflicted throttling.

When to escalate

Open a case with Autodesk support when you see: persistent 429s at modest call volumes after the fixes above; queries that stay slow even with simple id- or link-based filters; or webhook deliveries failing after configuration checks. First check the service status page for outages. Attach request logs with timestamps, your client library version, and a minimal reproduction script — support can do little with \"it's slow sometimes.\" Because the product was renamed, docs and UI labels may say either ShotGrid or Flow Production Tracking; search both names.