FastAPI BackgroundTasks vs Celery for jobs that may outlive the request timeout
0 reputation · 17 May 2022, 15:03 UTC
I'm designing an endpoint in a FastAPI service (running behind Uvicorn) that triggers follow-up work after returning a response — things like sending notifications and updating an external system. Some of these jobs are quick, but a few can run for several seconds and must not be lost if the server restarts.
FastAPI's built-in BackgroundTasks is attractive because it needs no extra infrastructure, but my understanding is that tasks run in the same process, offer no retry or timeout mechanism, and failures are not surfaced anywhere. Celery with a Redis or RabbitMQ broker would give durability, retries, and monitoring, at the cost of operating a broker and worker fleet.
The constraint: jobs must survive a deploy or restart, and the HTTP response must not be delayed by slow work.
Is it correct that a BackgroundTask that blocks for several seconds can delay the client response or hit proxy timeouts, or does Starlette send the response first in current versions? For guaranteed delivery, is Celery the only sensible option, or is there a documented lighter-weight pattern within FastAPI? If I mix both, what is a reasonable rule for deciding which jobs go where?