Diagnosing Stuck Background Jobs in NetBox's RQ Task Queue

The condition: jobs pile up and nothing runs

You notice it in one of three ways: webhooks that used to fire on object changes have gone silent, a custom script or report sits at pending forever, or the job list in the admin UI keeps growing while nothing completes. Redis itself answers connections, and the NetBox web frontend works fine. That combination almost always points at the background worker layer, not the application.

NetBox offloads webhooks, custom scripts, and reports to background workers built on RQ (Redis Queue). Jobs are enqueued in Redis by the web process and executed by a separate worker process — typically a systemd service such as netbox-rqworker on bare-metal installs, or a dedicated worker container in Docker deployments. If that process is down, crashing, or talking to the wrong Redis, jobs accumulate silently. The useful takeaway: diagnose the worker before touching the queue, because flushing Redis destroys pending webhook and script jobs you may want to keep.

Cause and diagnostic table

Ordered checks

1. Confirm Redis is reachable. From the NetBox host, run redis-cli -h <redis-host> ping (adjust for your deployment; no special privileges needed beyond network access). You expect PONG. If this fails, fix Redis first — nothing else matters until it answers.
2. Check the worker service. On a systemd install: sudo systemctl status netbox-rqworker and sudo journalctl -u netbox-rqworker -n 100 --no-pager. On Docker: docker compose ps and docker compose logs --tail=100 <worker-service>. A worker that starts and dies within seconds usually shows a Python ImportError or traceback naming a plugin — that is a code problem, not a Redis problem.
3. Inspect a stuck job in the UI. Open the job's detail page (under the script/report or webhook job list) and read its recorded status and traceback. A job with a traceback failed in your code or a plugin; a job with no traceback that never left pending was never picked up.
4. Run a trivial script. Create or run a minimal custom script that just logs a line, from the NetBox UI. If it transitions from pending to completed within seconds, the pipeline is healthy and the fault is in the specific failing job.
5. Review recent changes. Correlate the first stuck job's timestamp with the last upgrade, plugin install, or config edit. Upgrades are the classic trigger: the worker service is frequently left running old code or not restarted at all.

Fixes tied to findings

Worker not running: restart and enable it — sudo systemctl enable --now netbox-rqworker (systemd) — and make this a standard step after every NetBox upgrade. Verify with the trivial script test above.

Plugin import error: the traceback in the worker log names the module. Disable the offending plugin in configuration.py (remove it from PLUGINS), restart the worker, and confirm it stays up. Then fix or pin the plugin version before re-enabling.

Wrong Redis or queue config: compare the Redis settings the web process uses with what the worker sees — they must point at the same Redis instance, database, and queue names. A mismatch produces the confusing state where the worker runs happily while jobs pile up elsewhere. Correct the config and restart both sides.

Job-level failures: once the worker is healthy, re-run failed scripts or reports from the UI. Failed webhook jobs may need retriggering by repeating the originating change, depending on your version's requeue options — check before assuming automatic retry.

Avoid the tempting shortcut: do not flush Redis or purge queues as a first resort. That permanently deletes pending webhook and script jobs. If you must clear a poisoned queue, drain or requeue what you can first, and note exactly what you discarded.

Escalation criteria

Escalate upstream when, after a clean worker restart with a minimal configuration (plugins disabled, stock settings), jobs still fail with tracebacks inside core NetBox code rather than your scripts or plugins. Before filing an issue, capture: the full job traceback, the worker log around the failure timestamp, your exact NetBox version, deployment method, and plugin list. That package of evidence is what maintainers need, and collecting it also tells you whether the fault survives a minimal config — the line between "my deployment" and "a real bug."

Note: service names, unit files, and management commands vary by NetBox version and deployment method. Confirm names against your installed version's documentation before copy-pasting commands.