Webhook delivery jobs linger after HTTP timeout expires
0 reputation · 25 Jun 2026, 10:24 UTC
Background
NetBox v3.5+ uses RQ workers to deliver webhook payloads with a configurable WEBHOOK_TIMEOUT (default 10 seconds) for the outbound HTTP request. The same worker process honors RQ_JOB_TIMEOUT (default 300 seconds) as a hard limit on job execution time.
Observed gap
When a remote endpoint exceeds WEBHOOK_TIMEOUT, the HTTP client aborts and marks the delivery failed. However, the RQ job remains active until RQ_JOB_TIMEOUT elapses because the worker does not exit early after the HTTP layer times out. This leaves worker slots occupied for up to five minutes per timed-out delivery, reducing throughput for subsequent webhooks.
Constraints
The retry logic (WEBHOOK_RETRY_INTERVALS) treats timeout failures identically to other HTTP errors, so retries are scheduled normally. There is no API or UI action to cancel a dequeued webhook job, and the delivery log does not differentiate connection timeouts from read timeouts.
Questions
- Is the worker intended to terminate immediately once the HTTP client reports a timeout, or is the current behavior (waiting for
RQ_JOB_TIMEOUT) by design? - Can
RQ_JOB_TIMEOUTbe safely lowered to matchWEBHOOK_TIMEOUTwithout breaking other RQ-backed tasks? - Are there plans to surface timeout classification (connect vs. read) in the delivery status model?