ShotGrid Event Daemon vs Webhooks: Choosing the Right Automation Path for Your Pipeline

The problem: your pipeline automation is stuck on legacy architecture

Most studios running ShotGrid have an shotgunEventDaemon humming away on a Linux box somewhere — loading Python plugins that react to Shot changes, Version updates, and Note creation. It works, but it's a single point of failure, requires manual scaling, and Autodesk hasn't added features to it since webhooks went GA in 2021. Meanwhile, the modern webhook system delivers HTTPS POST payloads with HMAC-SHA256 signatures, exponential backoff retries, and per-webhook filtering. The question isn't whether to migrate; it's how to migrate without breaking the automations your artists depend on daily.

Architecture comparison: persistent connection vs. stateless delivery

The daemon maintains a long-lived AMQP connection to ShotGrid's event stream. Plugins run in a single Python interpreter, processing events sequentially. A crash in one plugin stops all event processing until restart. Webhooks, by contrast, are stateless: ShotGrid POSTs JSON to your endpoint with a 30-second response timeout. You acknowledge immediately and offload work to a queue (Celery, SQS, RQ). This shifts operational burden from "keep a Python process alive" to "run a scalable HTTP service."

Payload differences matter for implementation. Daemon plugins receive the full event — previous and new entity state — enabling multi-entity transactions in one callback. Webhooks deliver only the changed entity snapshot; you'll need follow-up API calls for related data. For a Version creation that should trigger a transcode, the daemon plugin sees the Version and its linked Task in one payload. The webhook handler gets the Version and must query /api/v1/entity/Task/... to find the upstream Task.

Reliability models: ordering guarantees vs. idempotency design

The daemon processes events sequentially per plugin, guaranteeing ordering for dependent operations. If Plugin A creates a filesystem folder and Plugin B writes a metadata file, they run in order. Webhooks offer up to 100 concurrent deliveries with configurable batch windows (1–60s). Ordering isn't guaranteed across deliveries. You handle this by designing handlers around the X-Shotgun-Event-Id header — every delivery carries a unique ID, and retries reuse the same ID. Your handler must be idempotent: check "have I processed this event ID?" before acting.

Retry behavior differs significantly. The daemon retries by re-processing the event stream on restart. Webhooks use exponential backoff up to 24 hours. If your endpoint returns 5xx or times out, ShotGrid will retry. A non-idempotent handler creates duplicate folders, duplicate transcodes, duplicate notifications. The X-Shotgun-Event-Id is your deduplication key; store it with a TTL matching the retry window.

Worked example: migrating a "Version → Transcode" plugin

Assume a daemon plugin that triggers an FFmpeg transcode when a Version's sg_uploaded_movie field changes. The plugin reads the Version entity, finds the linked Task, writes a job JSON to a watched directory, and a separate worker picks it up.

Step 1: wrap the logic in a FastAPI endpoint

from fastapi import FastAPI, Request, Header, HTTPException
import hmac, hashlib, os

app = FastAPI()
WEBHOOK_SECRET = os.environ["SHOTGRID_WEBHOOK_SECRET"]
PROCESSED_EVENTS = set()  # replace with Redis in production

@app.post("/shotgrid/version-transcode")
async def handle_version_transcode(
    request: Request,
    x_shotgun_event_id: str = Header(...),
    x_shotgun_signature: str = Header(...),
):
    body = await request.body()
    # Verify HMAC-SHA256
    expected = hmac.new(
        WEBHOOK_SECRET.encode(), body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, x_shotgun_signature):
        raise HTTPException(401, "Invalid signature")
    
    # Idempotency check
    if x_shotgun_event_id in PROCESSED_EVENTS:
        return {"status": "duplicate"}
    PROCESSED_EVENTS.add(x_shotgun_event_id)
    
    payload = await request.json()
    entity = payload.get("entity", {})
    if entity.get("type") != "Version":
        return {"status": "ignored"}
    
    # Offload to queue — acknowledge within 30s
    queue.enqueue(transcode_job, entity["id"])
    return {"status": "accepted"}

Step 2: create the webhook in ShotGrid

Use the Python API or web UI. Filter to Version entity, field sg_uploaded_movie, project scope as needed. Set the URL to your HTTPS endpoint (ngrok for local dev, real domain for staging/prod). The secret you generate becomes SHOTGRID_WEBHOOK_SECRET.

Step 3: handle the follow-up query

The webhook payload gives you the Version ID. Your queued transcode_job function queries ShotGrid for the Version's task field and any custom fields the transcode needs. This extra API call is the main migration cost — but it decouples your handler from ShotGrid's event payload structure.

Trade-offs and limitations you'll hit

• Air-gapped / on-prem render farms: Webhooks require outbound internet from ShotGrid Cloud to your endpoint. If your transcode farm has no internet path, the daemon remains your only option. Hybrid setups run both during transition.
• Complex filtering: Webhook filters use a simplified syntax (field=value, field__in=val1,val2). You can't express joins like "Version where Task's PipelineStep is 'comp'". You'll filter broadly in the webhook and refine in code.
• Daemon plugin crashes: One bad plugin stops all processing. Webhooks isolate failures per endpoint — a broken transcode handler doesn't break your Slack notification handler.
• Local development: You need a public tunnel (ngrok, Cloudflare Tunnel) or the CLI's shotgun webhook test command to simulate payloads. The daemon runs locally with no tunnel needed.
• No dead-letter queue: As of 2024, ShotGrid webhooks don't expose a dead-letter mechanism. Events that exhaust retries are lost. Monitor delivery success rates via the webhook UI.

Actionable next steps

1. Inventory your daemon plugins. List each plugin, its trigger entity/fields, and whether it needs multi-entity context. Prioritize stateless, single-entity plugins for first migration.
2. Spin up a dev webhook target. Use ngrok http 8000 and the ShotGrid CLI test command to verify payload structure, HMAC verification, and idempotency handling.
3. Add a queue layer. Even if you process synchronously today, put a queue (Redis + RQ, Celery, SQS) between the webhook endpoint and the work. The 30-second timeout is non-negotiable.
4. Run both in parallel. Create the webhook alongside the daemon plugin. Compare event latency, duplicate rates, and missed events for two weeks before cutting over.
5. Plan secret rotation. Webhook HMAC secrets only rotate when you recreate the webhook. Build a rotation procedure: create new webhook → update secret in deployment → verify → delete old webhook.

The daemon isn't deprecated today, but it's in maintenance mode. Every new pipeline automation you write should target the webhook model. The migration effort pays off in operational simplicity: stateless services scale horizontally, fail independently, and don't require a dedicated VM babysitting a Python process.