uWSGI Spooler: Built-in Background Jobs Without Extra Infrastructure

The Problem: Offloading Work Without Adding Dependencies

Web applications routinely need to fire off work that doesn't belong in the request/response cycle — sending email, generating PDFs, retrying webhooks, or running ML inference. The typical answer is a message broker (Redis, RabbitMQ, Kafka) plus a worker fleet. That adds operational surface: more services to deploy, monitor, secure, and scale. uWSGI's Spooler subsystem offers a different trade-off: a file-based, at-least-once job queue that lives inside the application server you're already running. No extra processes, no network hop, no new credentials to rotate. The catch is you accept its constraints: single-host scaling, no priorities or TTL, and fsync-bound throughput.

Requirements It Satisfies

• Move long-running or unreliable work out of HTTP workers so request latency stays predictable.
• Survive process restarts — tasks persist on disk.
• Zero additional infrastructure (no Redis, no RabbitMQ).
• At-least-once delivery is acceptable; exactly-once is not required.

If you need strict ordering, multi-host horizontal scaling, or >1,000 tasks/second, the Spooler is the wrong tool — plan for an external queue from the start.

Smallest Suitable Design

Define a spool directory owned by the application user, enable the spooler in your uWSGI configuration, and write a spooler callable that returns a status code.

Configuration (uWSGI ini)

[uwsgi]
# ... your existing config ...
spooler = /var/spool/uwsgi/tasks
spooler-processes = 2
spooler-frequency = 3
stats = :9191

spooler is the on-disk queue directory. spooler-processes sets how many dedicated spooler workers uWSGI spawns (separate from HTTP workers). spooler-frequency is the poll interval in seconds — default 30s; drop to 3s for lower latency at the cost of more filesystem scans. stats exposes a JSON endpoint for monitoring.

Directory Setup

# Run as root or via provisioning
mkdir -p /var/spool/uwsgi/tasks
chown www-data:www-data /var/spool/uwsgi/tasks
chmod 700 /var/spool/uwsgi/tasks

Restrict permissions to the app user only. The spool directory is trusted storage — no encryption or signing is built in.

Enqueueing from Application Code

import uwsgi

def send_welcome_email(user_id: int):
    uwsgi.spool({
        'task': 'send_email',
        'template': 'welcome',
        'user_id': user_id,
        'retry_count': 0
    })

uwsgi.spool() accepts a JSON-serializable dict. It blocks until the task file is fsynced to disk — if the filesystem is full, the call blocks (ENOSPC). For latency-sensitive paths, consider a circuit breaker or async enqueue wrapper.

The Spooler Callable

import uwsgi
from myapp.email import render_and_send
from myapp.models import User

@uwsgi.spooler
def process_spool_task(args: dict) -> int:
    task = args.get('task')
    if task == 'send_email':
        user = User.objects.get(id=args['user_id'])
        render_and_send(args['template'], user.email)
        return 0  # done, delete spool file
    
    # Unknown task — discard to avoid infinite retries
    return -1

The decorator registers the function as the spooler entry point. Return values control the queue:

• 0 — success, spool file deleted.
• -1 — discard permanently (poison pill, invalid payload).
• -2 — retry later; file stays, re-picked after spooler-frequency.

Critical: uWSGI does not track retry counts. A task that always returns -2 retries forever. Implement your own counter in the payload and return -1 after a threshold.

Trust and Data Boundaries

Spool files are plain JSON written by HTTP workers and read by spooler workers. There is no cross-host transport or authentication between producer and consumer. Treat the spool directory as trusted storage:

• Validate and sanitize every field inside the spooler function.
• Restrict filesystem permissions to the app user (700).
• If you run multiple apps under Emperor mode, give each vassal its own subdirectory (e.g., /var/spool/uwsgi/app1/tasks) to avoid cross-app task collision.

Operational Checks

Queue Depth

# Run on the application host
find /var/spool/uwsgi/tasks -type f | wc -l

Prometheus Metrics via Stats Endpoint

curl -s localhost:9191 | jq '.spooler_queue_size, .spooler_busy_workers'

Scrape these into Prometheus. spooler_queue_size is the number of pending task files; spooler_busy_workers shows how many spooler processes are actively executing tasks.

Disk Health

df -h /var/spool/uwsgi
df -i /var/spool/uwsgi

Disk full or inode exhaustion stops enqueueing — HTTP workers block on spool().

Failure Modes and Mitigations

Failure	Behavior	Mitigation
Spooler worker crashes mid-task	Task file remains; re-picked on restart (at-least-once)	Design tasks to be idempotent
Poison pill (always returns -2)	Infinite retries, growing backlog	Implement retry counter in payload; return -1 after N attempts
Disk full (ENOSPC)	uwsgi.spool() blocks; HTTP workers stall	Monitor disk; circuit breaker around enqueue
Spooler OOM	Master respawns process; in-flight task lost unless idempotent	Limit task memory; consider limit-as in config
Large payloads (>1 MB)	Spool files bloat disk, fsync slows	Store blobs externally; put reference in spool payload

Conditions That Change the Design

• Strict ordering or exactly-once required — add Redis Streams or Kafka.
• Multi-host scaling — shared NFS works but introduces latency and lock contention; prefer external queue.
• High throughput (>1k tasks/sec) — fsync overhead becomes bottleneck.
• Payloads >1 MB — use external blob store + reference.
• Security zones requiring network isolation — spooler must run in same trust zone as producers.

Async Functions and Greenlets

The spooler does not support Python async def functions directly. If your worker uses gevent or greenlet, the spooler runs in that context. For native asyncio code, wrap the call: asyncio.run(coro()) inside a synchronous spooler function.

Verification Checklist

1. Create a minimal ini with spooler = /tmp/uwsgi_spool, spooler-processes = 1, stats = :9191.
2. Start uWSGI.
3. From a Python shell: import uwsgi; uwsgi.spool({'test': 1}).
4. Observe a file appear in /tmp/uwsgi_spool and disappear after processing.
5. Query curl localhost:9191 | jq .spooler_queue_size — should return 0 after processing.
6. Simulate failure: have spooler function raise an exception; verify file persists and is retried.

Limitations Summary

• No priorities, TTL, visibility timeouts, or delayed execution.
• No horizontal scaling across hosts without shared filesystem.
• Default 30s poll interval; lower values increase CPU/inotify load.
• uwsgi.spool() blocks on full disk.
• No built-in retry counting — implement yourself.
• Emperor mode requires per-vassal spool directories.