Diagnosing and Fixing NetBox Webhook Delivery Failures

Recognizing Webhook Delivery Failures

NetBox webhooks are asynchronous; they are queued by the web application and processed by a background worker (RQ worker). A failure is typically recognized when the Webhook UI (/extras/webhooks/) shows entries in a queued or failed state, or when the retry count increases without the downstream consumer (such as a Slack bot, Nautobot, or custom API) receiving the payload.

Root Cause Diagnostic Table

Symptom	Likely Cause	Diagnostic Indicator
Immediate 'failed' status	TLS/SSL Verification Failure	Logs show SSLError or CertificateVerifyFailed
Stuck in 'queued'	RQ Worker Down	systemctl status netbox-rqworker is inactive
Timeout errors	Receiver Latency	Logs show ReadTimeout or ConnectTimeout
401/403 Unauthorized	Signature Mismatch	Receiver reports invalid HMAC-SHA512 signature
413 Payload Too Large	Receiver Body Limit	Receiver logs show Request Entity Too Large
415 Unsupported Media	Content-Type Mismatch	Receiver expects JSON but receives form-encoded data

Ordered Verification Steps

Follow these steps in order to isolate whether the failure is internal to NetBox, a network issue, or a receiver-side configuration error.

1. Verify the Background Worker

Since webhooks rely on Redis Queue (RQ), the worker process must be running. If the worker is dead, webhooks will remain in the queued state indefinitely.

# For systemd installations
systemctl status netbox-rqworker

# For Docker installations
docker ps | grep rqworker

2. Inspect Worker Logs

The RQ worker logs provide the exact exception thrown during the HTTP request. Look for traceback errors related to requests or ssl.

# For systemd
journalctl -u netbox-rqworker -n 200

# For Docker
docker logs netbox-rqworker

3. Test Network Reachability

Run a manual curl command from the NetBox host to the receiver endpoint to rule out firewall or routing issues. Use a dummy JSON payload.

# Run from the NetBox application server
curl -v -X POST -H 'Content-Type: application/json' -d '{}' https://<receiver-endpoint>/

4. Validate TLS Chain

If the receiver uses a self-signed certificate, verify the chain from the NetBox host to see if the OS trusts the CA.

openssl s_client -connect <receiver-host>:443 -showcerts </dev/null

Targeted Fixes

Fixing SSL/TLS Failures

If the receiver uses a self-signed certificate and you cannot add the CA to the system trust store, you can disable global verification in configuration.py. Warning: This disables SSL verification for all webhooks, increasing vulnerability to man-in-the-middle attacks.

# In configuration.py
NETBOX_WEBHOOKS_VERIFY_SSL = False

Resolving Timeouts

If the receiver is slow to process the payload (e.g., triggering a heavy automation script), increase the timeout. The default is 10 seconds.

# In configuration.py
NETBOX_WEBHOOKS_TIMEOUT = 30

Correcting Signature Mismatches

If the receiver rejects the X-Hook-Signature, ensure the shared secret in the NetBox Webhook UI matches the secret configured in the receiver's HMAC-SHA512 calculation. If rotating secrets, update the receiver first, then NetBox, to avoid a window of total failure.

Handling Content-Type Mismatches

NetBox v4.0+ defaults to JSON payloads. If you are using a legacy receiver that expects form-encoded data, you must either upgrade the receiver or use a custom header override in the Webhook configuration to signal the correct format.

Verification and Rollback

To verify the fix, use the Test button within the NetBox Webhook UI. This triggers a dummy payload. Confirm a 2xx response in the receiver logs and a 'success' state in the NetBox UI.

To check for a backlog of failed webhooks in the database, run the following via the NetBox management shell:

# Run as the netbox user
netbox manage shell -c "from extras.models import Webhook; print(Webhook.objects.filter(status='failed').count())"

Rollback: If changes to configuration.py (such as NETBOX_WEBHOOKS_VERIFY_SSL) cause unexpected behavior, revert the value and restart the NetBox services and the RQ worker: systemctl restart netbox netbox-rqworker.

Escalation Criteria

Escalate to system administrators or security teams if any of the following occur:

• OOM Kills: The RQ worker is repeatedly killed by the kernel (check dmesg | grep -i oom).
• Queue Bloat: The extras_webhook table exceeds 10,000 rows, indicating a massive backlog that may impact PostgreSQL performance.
• Persistent 5xx: The receiver returns 5xx errors for more than 10 consecutive retries despite network reachability.
• Compliance Conflict: Security policy mandates TLS verification, but the receiver cannot provide a valid, trusted certificate.