Building a Reliable, Idempotent Webhook Pipeline for Sanity.io

Problem Statement

Sanity.io fires HTTP POST webhooks whenever a document changes. In production, a single lost delivery can break downstream workflows, and duplicate deliveries can corrupt state. The challenge is to design a system that guarantees at‑least‑once delivery, preserves idempotency, and protects against replay attacks, all while remaining operationally simple.

Requirements

• At‑Least‑Once Delivery: Every change must reach the consumer, even if the consumer is temporarily down.
• Idempotency: Re‑sending the same event should not alter the downstream state.
• Security: Only legitimate Sanity webhooks should be accepted; the payload must be authenticated.
• Operational Visibility: Metrics on success/failure rates, queue depth, and retry counts.
• Graceful Failure Handling: Network hiccups, consumer crashes, or payload changes should not cause data loss.

Minimal Suitable Design

The simplest architecture that satisfies the above is a two‑step pipeline:

1. Sanity Webhook → Queue – An HTTP endpoint receives the webhook, verifies the signature, and pushes the payload into a durable queue (RabbitMQ, AWS SQS, or Azure Service Bus).
2. Queue → Consumer – A worker (e.g., Lambda, Docker container, or server process) pulls messages, deduplicates using the _rev field, and forwards the payload to the final destination.

Figure 1 illustrates the flow.

Trust & Data Boundaries

Sanity → Queue
The HTTP endpoint must be isolated from the rest of the system. It should only expose a single /webhook path, accept POST requests, and perform the following checks:

• Validate the X‑Sanity‑Signature header using the shared secret (HMAC‑SHA256).
• Reject any request without the header or with a mismatched signature.
• Return 200 OK immediately after enqueuing to avoid client timeouts.

Because the queue is the only point of contact with downstream services, the consumer can assume all messages are authentic.

Queue → Consumer
The consumer should treat each message as immutable. The _rev field is a unique revision identifier; it can be used to deduplicate. The consumer must keep a small cache or a persistent store (e.g., Redis, DynamoDB) keyed by documentId + _rev to detect duplicates.

Operational Checks

• Health‑Check Endpoint – /health returns 200 OK if the queue client can connect.
• Metrics – Expose counters for enqueued, processed, failed, and duplicate events. Use Prometheus or CloudWatch.
• Alerting – Trigger alerts when queue depth exceeds a threshold or when the failure rate rises above 1%.
• Dead‑Letter Queue (DLQ) – After a configurable number of retries, move the message to a DLQ for manual inspection.

Concrete Example: AWS SQS + Lambda

Below is a minimal implementation using AWS services. Replace <SANITY_SHARED_SECRET> with your secret.

# Lambda function (Python 3.11)
import os
import hmac
import hashlib
import json
import boto3

sqs = boto3.client('sqs')
QUEUE_URL = os.environ['QUEUE_URL']
SHARED_SECRET = os.environ['SANITY_SHARED_SECRET'].encode()


def lambda_handler(event, context):
    # Sanity webhook body is JSON
    body = json.loads(event['body'])
    signature = event['headers'].get('X-Sanity-Signature')
    if not signature:
        return {'statusCode': 400, 'body': 'Missing signature'}
    # Verify HMAC
    expected = hmac.new(SHARED_SECRET, event['body'].encode(), hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        return {'statusCode': 403, 'body': 'Invalid signature'}
    # Enqueue
    sqs.send_message(QueueUrl=QUEUE_URL, MessageBody=event['body'])
    return {'statusCode': 200, 'body': 'Queued'}

The consumer Lambda (or a long‑running process) pulls messages from the SQS queue, checks the _rev against Redis, and forwards the data. A retry policy in SQS (e.g., 5 attempts with exponential back‑off) ensures at‑least‑once delivery.

Failure Modes & Mitigations

• Network Timeouts – The webhook endpoint should return 200 before the queue operation completes. If the queue write fails, the Lambda can retry or send to DLQ.
• Consumer Downtime – SQS visibility timeouts prevent message loss. If the consumer is down, messages remain in the queue until processed.
• Replay Loops – The deduplication cache must be durable. If the cache is lost, a duplicate _rev may be processed again. Use a persistent store for high reliability.
• Payload Evolution – If Sanity adds fields, the consumer should be tolerant (ignore unknown keys). If the _rev format changes, the deduplication logic must be updated.

When to Change the Design

• When downstream services require strict ordering beyond idempotency. In that case, add a sequence number or use a partitioned queue.
• When the volume exceeds what a single queue can handle. Scale horizontally by sharding the queue or adding more consumers.
• When the consumer cannot tolerate duplicate _rev values because the business logic is not idempotent. Add stateful processing or compensating actions.
• When Sanity changes the signature algorithm or introduces a new authentication header. Update the verification logic accordingly.

Practical Verification Steps

1. Deploy the webhook endpoint and expose it via a public URL (e.g., API Gateway).
2. Configure Sanity to point to this URL under project settings → Webhooks.
3. Trigger a document change and inspect the queue metrics to confirm the message appears.
4. Simulate a consumer crash by temporarily disabling the consumer Lambda. Verify that the message stays in the queue and is retried.
5. Send the same change again to test idempotency. The consumer should log a duplicate and skip processing.

Each step should be logged and monitored. If any verification fails, adjust the configuration before moving to production.

Conclusion

By decoupling the Sanity webhook from the consumer through a durable queue and implementing HMAC verification plus _rev‑based deduplication, you achieve a robust, secure, and operationally visible pipeline. The design is minimal yet extensible; it can evolve to meet ordering guarantees or higher throughput without breaking the core guarantees.