Building Reliable Pipeline Automation with the ShotGrid Event Daemon

When automating a production pipeline, the biggest risk isn't a bug in your logic—it's a network timeout or a server crash happening exactly halfway through a critical operation. If you rely on simple webhooks to trigger filesystem creation or status propagation, a single dropped packet can leave a ShotGrid entity out of sync with your actual storage, forcing manual cleanup across hundreds of shots.

The ShotGrid Event Daemon solves this by moving from a "push" model (webhooks) to a "pull" model. Instead of waiting for ShotGrid to tell your server something happened, the daemon polls the EventLog, tracks its own progress in a local database, and ensures every event is processed at least once.

The Architecture of Reliability

The Event Daemon is a long-lived Python service that acts as a dispatcher. It monitors the ShotGrid EventLog via the Python API and routes events to specific plugins based on the entity type (e.g., Shot, Task) and the event type (e.g., Change, Create).

At-Least-Once Delivery and Idempotency

The daemon maintains a monotonic cursor in a local SQLite state database. This cursor records the ID of the last successfully processed event. If the daemon crashes or the server reboots, it doesn't start from scratch; it resumes from the last recorded ID.

Because a crash can occur after a plugin executes but before the cursor is updated, the daemon may deliver the same event twice. This makes idempotency mandatory. An idempotent plugin is one where running the same operation multiple times has the same result as running it once. For example, instead of "Append 'Approved' to a log file," use "Ensure the Shot status is set to 'Approved' in the database."

Retry Semantics and the Dead-Letter Queue

Network flakiness is inevitable. The daemon implements exponential backoff for failed plugins. If a plugin fails, the daemon retries the operation with increasing delays. If the maximum retry count is reached, the event is moved to a dead-letter queue in the local SQLite DB. This prevents a single corrupted entity or a broken API call from blocking the entire pipeline queue.

Implementation: A Status Propagation Plugin

A common requirement is to update a parent entity when a child is completed. In this example, when a Task is marked as "Complete," the plugin checks if all sibling tasks are done and updates the parent Shot status.

# Example Plugin Logic (Conceptual)
# This would be wrapped in the ShotGrid Event Daemon plugin class structure

def execute(self, event):
    entity_type = event['entity_type']
    event_type = event['event_type']
    entity_id = event['entity_id']
    
    # Only trigger on Task changes
    if entity_type == 'Task' and event_type == 'Change':
        # Check if 'sg_status_list' was actually changed to 'cmplt'
        if event['changed_fields'].get('sg_status_list') == 'cmplt':
            parent_shot = self.sg.find_one('Shot', [['id', 'is', event['parent_id']]])
            
            # Idempotent check: only update if not already 'Final'
            if parent_shot['sg_status_list'] != 'final':
                self.sg.update('Shot', parent_shot['id'], {'sg_status_list': 'final'})
                self.logger.info(f"Updated Shot {parent_shot['id']} to Final")

Execution Context: This code runs within the daemon process on your studio server. It requires a ShotGrid API script user with write permissions to both the Task and Shot entities.

Operational Trade-offs: Daemon vs. Webhooks

Feature	Event Daemon	Webhooks
Delivery Guarantee	At-least-once (via local state)	Best effort (push)
Network Setup	Outbound only (Secure)	Requires public ingress/HTTPS
Latency	Poll interval (e.g., 5s)	Near real-time
Scalability	Single instance per site	Horizontally scalable

Limitations and Risks

• API Rate Limits: On ShotGrid Cloud, aggressive polling (e.g., every 1 second) combined with heavy plugins can trigger 429 Too Many Requests. Monitor the X-RateLimit-Remaining headers in your logs.
• Single Point of Failure: The daemon is not horizontally scalable. Running two instances against the same site will result in duplicate plugin executions. Use systemd or supervisord to ensure the process restarts automatically on failure.
• Database Bloat: The SQLite state DB grows over time. For high-volume sites, implement a monthly VACUUM or archive job to prevent slow restart times.

Verification Checklist

To verify your daemon is healthy, run the following checks:

1. Cursor Check: Run sqlite3 event_daemon.db "SELECT * FROM processed_events ORDER BY id DESC LIMIT 1;". The ID should increment as you make changes in ShotGrid.
2. Failure Test: Manually trigger a plugin failure (e.g., by temporarily changing a field name in the code). Verify in the logs that the daemon attempts retries with increasing intervals before moving the event to the dead-letter queue.
3. Latency Audit: Compare the timestamp of a ShotGrid event creation with the plugin_duration_ms in the JSON logs to ensure your poll interval isn't creating a bottleneck.