Offloading Background Work with uWSGI's Built-in Spooler

The problem: HTTP workers stuck on slow tasks

Every web application eventually needs to do work that takes longer than a request timeout — generating PDFs, sending emails, calling external APIs, or running ML inference. The naive approach blocks an HTTP worker, reducing capacity and increasing latency for every other request. External queues like Celery or RQ solve this, but they add infrastructure: a broker (Redis, RabbitMQ), separate worker processes, and operational overhead.

uWSGI includes a built-in background job queue called the Spooler. It writes jobs as files to a directory, and dedicated spooler processes pick them up. No extra services, no new dependencies, and it survives uWSGI restarts. It's not a replacement for distributed task queues, but for low-to-moderate volume on a single host (or a few hosts with shared storage), it's often the simplest path from "this is too slow" to "this runs in the background."

How the Spooler works

The Spooler is a directory watched by one or more spooler processes. When your application calls uwsgi.spool(), uWSGI serializes a job dictionary to a file in that directory. Each spooler process scans the directory every --spooler-frequency seconds (default 30), picks up to --spooler-max-tasks files per scan, deserializes them, and executes the named callable in the same Python interpreter.

Jobs are plain dictionaries with reserved keys:

• callable — dotted path to a function (e.g., 'tasks.send_email')
• args / kwargs — positional and keyword arguments
• priority — integer, lower runs first
• at — unix timestamp for delayed execution
• retry — seconds before re-queue on failure

If the callable raises, the job file moves to a .retry subdirectory and is retried after the retry interval. After --spooler-max-retries (default 3) it lands in .failed for manual inspection.

Configuration and startup

Add spooler options to your uWSGI ini or command line:

[uwsgi]
spooler = /var/spool/uwsgi/myapp
spooler-processes = 2
spooler-frequency = 10
spooler-max-tasks = 100
spooler-max-retries = 3
# optional recurring jobs
spooler-cron = 0 2 * * * tasks.nightly_cleanup

The spool directory must exist and be writable by the uWSGI master/user. Use a local filesystem with good metadata performance — avoid NFS for high throughput because each job is a separate file (inode exhaustion is real at scale).

Worked example: PDF generation endpoint

Here's a minimal Flask-style app showing the pattern. The HTTP endpoint validates input, enqueues a minimal payload, and returns 202 Accepted immediately. The spooler worker does the heavy lifting.

# app.py
import uwsgi
from flask import Flask, request, jsonify

app = Flask(__name__)

def generate_report(user_id, report_type):
    """Heavy work: render PDF, upload to S3, notify user."""
    # ... your logic here ...
    pass

@app.route('/reports', methods=['POST'])
def create_report():
    data = request.get_json()
    user_id = data.get('user_id')
    report_type = data.get('type')
    if not user_id or not report_type:
        return jsonify(error='missing fields'), 400
    
    # Enqueue minimal payload — pass IDs, not objects
    uwsgi.spool({
        'callable': 'app.generate_report',
        'args': [user_id, report_type],
        'priority': 10,
        'retry': 60  # retry after 60s on failure
    })
    return jsonify(status='queued'), 202

if __name__ == '__main__':
    app.run()

Start uWSGI with the spooler config above. When you POST to /reports, a file like /var/spool/uwsgi/myapp/1700000000-12345 appears. The spooler process deserializes it and calls generate_report(user_id, report_type).

Trade-offs and limitations

The Spooler's simplicity comes with constraints you should understand before committing:

• No horizontal scaling across hosts without a shared filesystem (and even then, no locking — two spoolers on different hosts can pick the same job).
• No built-in metrics — queue depth, latency, throughput require custom tooling (watch directory size, tail logs, or build a Prometheus exporter).
• Memory duplication — each spooler process loads your full application. If you import a 2 GB ML model at module level, --spooler-processes 4 means ~8 GB RAM just for spoolers. Consider mule processes or external workers for heavy dependencies.
• Pickle-only serialization — job payloads use uWSGI's serializer (pickle-based). Avoid non-picklable objects: database connections, file handles, lambdas, closures. Pass IDs and let the worker fetch fresh objects.
• No deduplication or priority inversion protection — duplicate submissions create duplicate work. Idempotency is your responsibility.
• No task chaining, rate limiting, or result backends — if you need workflows, Celery/RQ remain the right choice.

When to reach for the Spooler

Use the Spooler when:

• You run on one or two hosts and can accept single-host queue semantics.
• Volume is low-to-moderate (hundreds to low thousands of jobs/day).
• You want zero additional infrastructure and operational simplicity.
• Jobs are fire-and-forget or write results to a database/object store.

Switch to Celery/RQ when you need distributed workers, multi-language support, task chaining, visibility APIs, or rate limiting.

Verify it works in your environment

1. Run uwsgi --help | grep -i spool to confirm your build includes spooler options.
2. Create the spool directory: mkdir -p /var/spool/uwsgi/myapp && chown www-data:www-data /var/spool/uwsgi/myapp.
3. Start uWSGI with the config above.
4. Trigger the endpoint and watch ls -la /var/spool/uwsgi/myapp/ — job files should appear and disappear as they're processed.
5. Test failure handling: raise an exception in generate_report and verify .retry then .failed directories populate after the configured intervals.
6. Check memory: ps aux | grep spooler before and after loading heavy imports to confirm per-process duplication.

The Spooler isn't magic, but for the right scope it removes an entire moving part from your architecture. Start here, measure, and migrate only when the limitations bite.