Stop Pulling Full ShotGrid Records: Field Selection and Pagination Patterns That Actually Scale

Your nightly sync script worked fine when the project had 3,000 shots. Now it has 40,000, the script takes twenty minutes, and ShotGrid's rate limiter is throttling you halfway through. The fix usually isn't more hardware or clever threading — it's that the script is asking the API for far more than it needs, one oversized page at a time.

This post covers the two highest-leverage changes you can make to a shotgun_api3-based tool: selecting only the fields you actually read, and paginating deliberately instead of trusting defaults. Examples assume shotgun_api3 3.x against a hosted ShotGrid site; verify behavior against your own site before production use, since some limits are configurable per site.

The default query is quietly expensive

A call like sg.find('Shot', filters, ['code']) looks harmless, but two defaults work against you:

• Default page size is small. Historically the default limit is 20 records. A 40,000-shot export at 20 per page is 2,000 requests — and script keys are typically rate-limited on the order of ~100 requests per minute. You do the math on how that ends.
• Unrequested fields still cost you if you over-select. The inverse mistake is passing a huge field list (or helper code that requests "everything") — every extra column inflates the payload and server-side query time. Multi-entity fields like tasks are especially heavy.

The result is a script that's slow for reasons that have nothing to do with your data volume.

Ask for exactly the columns you read

The fields parameter is the cheapest optimization available. If your report only needs the shot code, status, and sequence, say so:

fields = ['id', 'code', 'sg_status_list', 'sg_sequence.Sequence.code']

shots = sg.find(
    'Shot',
    [['project', 'is', {'type': 'Project', 'id': project_id}],
     ['sg_status_list', 'in', ['ip', 'fin']]],
    fields,
    limit=1000,
    order=[{'field_name': 'code', 'direction': 'asc'}],
)

Run this wherever your script normally runs — a pipeline machine with the shotgun_api3 package installed and a script key whose Script entity has read permission on Shot. Nothing here changes state, so it's safe to experiment with.

Note the dot-notation field sg_sequence.Sequence.code: it traverses the sequence link and returns the sequence's code inline, saving you a second query per shot. This is one of the most underused features of the API. The trade-off is that deeply chained traversals make the server-side query heavier, so traverse one hop, not five.

A practical check: time the query with your full field list versus the minimal one on a real project. The difference is usually large enough to see without a profiler. Also confirm each field name still exists in your schema — ShotGrid doesn't version the API against schema changes, and a renamed custom field can silently break or empty a query depending on how the server handles it.

Paginate deliberately, and know your site's real cap

The limit parameter has a hard maximum — commonly 5,000 records per request, but it's a site-level setting and some studios lower it. Don't hard-code 5,000 into your loop. Verify your effective cap first:

probe = sg.find('Shot', [], ['id'], limit=10000)
print(len(probe))  # compare against your expected total

If the returned count is capped well below your known total, that's your effective page size. (On some configurations an over-limit request errors instead of truncating — either outcome tells you the cap.)

For large exports, later 3.x releases of the API ship shotgun_api3.lib.shotgun.find_iter(), a generator that handles the offset bookkeeping for you:

from shotgun_api3.lib.shotgun import find_iter

for shot in find_iter(sg, 'Shot', filters, fields, limit=1000):
    process(shot)  # stream instead of materializing 40k dicts

Check hasattr(sg, 'find_iter') or the module contents in a REPL before relying on it — older pinned versions of shotgun_api3 in studio deployments often predate it. If you're stuck on an older version, a manual loop incrementing offset by your page size until a short page comes back is the equivalent pattern.

One caveat: find_iter() fetches pages sequentially and still respects rate limits. It removes bookkeeping bugs, not latency. Don't try to parallelize page fetches across threads to compensate — you'll just trip the throttler and make everything slower.

Filter gotchas that produce wrong data, not errors

The dangerous bugs in ShotGrid queries are the silent ones:

• Multi-entity link fields need entity dicts. Filtering ['tasks', 'in', ...] requires a list of {'type': 'Task', 'id': ...} dicts. Pass raw integer IDs and you get zero results with no error.
• Datetimes are UTC. The server stores timestamps in UTC; the web UI displays them in the project's timezone. A between filter on created_at using local-midnight boundaries will produce off-by-hours (or off-by-day) report discrepancies. Convert to UTC ISO strings in your script, and validate by creating a test record with a known timestamp and querying it back.
• Complex logic needs the dict syntax. The legacy flat filter list ANDs everything. For grouped OR/AND logic, use the newer filters dictionary with logical_operator and conditions (API v3.3+). Confirm the current schema in the official Autodesk ShotGrid Python API reference before migrating — and test operator behavior against a known entity with the web UI's Advanced Search first.

The trade-off worth naming

Minimal field selection has a real cost: your code becomes coupled to an explicit schema contract. When someone renames sg_status_list's display name nothing breaks, but if a custom field is retired, your script fails or returns gaps rather than degrading gracefully. Mitigate this by centralizing field lists in one config module per tool, and add a startup sanity query that requests your field list against a single known record — a missing field surfaces immediately instead of halfway through a nightly batch.

What to do Monday morning

Pick your slowest ShotGrid script and make three changes: replace its field list with only the columns it reads, probe your site's effective page cap and set limit accordingly, and switch the export loop to find_iter() (or a manual offset loop) so results stream instead of accumulating. Then re-run it against a production-scale project and compare wall time. In most pipelines that combination turns a twenty-minute, throttle-hitting job into something that finishes quietly in a couple of minutes — no new infrastructure required.