Laravel Queue Architecture: Designing Resilient Job Processing with Redis and Horizon

The Problem: Background Work That Survives Failures

Sending emails, generating reports, and calling third-party APIs inside the request-response cycle makes pages slow and requests fragile. Moving that work to a queue solves the latency problem, but introduces new ones: workers that leak memory over days of uptime, jobs that vanish when a worker crashes, and retries that hammer an already-struggling external API. This note covers a minimal production-ready design for Laravel queues, the trust boundaries involved, and the conditions that would push you toward a different design.

Requirements

• Jobs must survive a worker crash and be retried with sensible limits.
• Workers must not degrade over time (memory growth, stale state).
• Operators must be able to see queue depth, throughput, and failures without SSHing into servers.
• The design should run on one or a few application servers without exotic infrastructure.

The Smallest Suitable Design

Laravel's queue system decouples dispatching from execution through a driver-based architecture. The three realistic drivers are database, redis, and sqs. For a typical production app, Redis is the right default: workers claim jobs with atomic pop operations, so multiple workers can poll the same queue without the row-locking contention that makes the database driver unsuitable under concurrency.

Configuration (in .env and config/queue.php, applied with php artisan config:cache after changes):

# .env
QUEUE_CONNECTION=redis

# config/queue.php
'redis' => [
    'driver' => 'redis',
    'connection' => 'default',
    'queue' => env('REDIS_QUEUE', 'default'),
    'retry_after' => 90,
    'block_for' => null,
],

retry_after must be larger than your longest job timeout. If a job runs longer than retry_after seconds, another worker can pick it up while it is still executing, causing duplicate work.

Worker Lifecycle and Memory Boundaries

Queue workers are long-running CLI processes. Because PHP objects, resolved container bindings, and framework state accumulate, a worker that runs for weeks will grow. The standard mitigation is to bound each worker's lifetime:

php artisan queue:work redis --tries=3 --timeout=60 --max-jobs=500 --sleep=3

Run this under Supervisor (or systemd) so the process is restarted after it exits. Each flag matters: --tries caps retry attempts, --timeout kills hung jobs, --max-jobs forces periodic restarts to reset memory, and --sleep controls polling frequency when idle. Run this as your web user (for example www-data) from the application root; workers need the same code, environment file, and PHP version as the web process.

Trust boundary: jobs are serialized PHP objects stored in the broker. The worker must have the exact job class available at the same namespace, or unserialization fails. This means deployments must be coordinated: never delete or rename a queued job class while jobs referencing it may still be pending. A rolling deploy that briefly runs old and new code side by side is safe only if both versions can handle the payload shape.

Keeping Payloads Small

Pass identifiers, not data. A job that receives an Eloquent model via SerializesModels re-fetches it from the database on execution, which keeps payloads tiny — but the model may have changed or been deleted by the time the job runs (handle ModelNotFoundException or use deleteWhenMissingModels). Passing large arrays or file contents into the constructor inflates the broker and worker memory. Store large inputs on disk or in object storage and pass a path or key instead.

Failure Handling and Backoff

Transient failures — API timeouts, rate limits — should be retried with delay rather than immediately. Configure this on the job class:

class SyncExternalOrder implements ShouldQueue
{
    public $tries = 5;
    public $backoff = [10, 60, 300, 900]; // seconds between attempts

    public function handle(): void
    {
        // call external API
    }
}

After the final attempt, the job lands in the failed_jobs table (create it with php artisan queue:failed-table and migrate). Inspect with php artisan queue:failed and retry with php artisan queue:retry <uuid>. To verify backoff behavior in a test environment, throw an exception inside a job and check that the job is reserved again only after the expected delay.

Operational Visibility with Horizon

For Redis-backed queues, Laravel Horizon provides a dashboard with throughput, wait times, failed jobs, and per-queue worker counts, plus automatic balancing of worker processes across queues. Install it, publish its config, and run a single supervised process:

php artisan horizon

Horizon manages the underlying workers itself — do not also run manual queue:work processes against the same queues. Protect the /horizon route via the Horizon::auth() gate, since it exposes job payloads and failure details.

Operational Checks

1. Dispatch a test job and run php artisan queue:work redis --once locally or on staging; confirm it completes and leaves the queue.
2. Confirm retry_after > timeout in config.
3. Watch worker memory over an hour: ps -o pid,rss,cmd -C php; RSS should reset when --max-jobs triggers a restart.
4. Throw a deliberate exception in a job and confirm it appears in queue:failed after the configured tries.
5. Load /horizon and verify worker counts and recent jobs appear.

Failure Modes and What Would Change the Design

• Duplicate side effects on retry. If a job charges a card or sends an email, retries must be idempotent — use unique constraints, idempotency keys, or check-before-act logic. If you cannot make a job idempotent, queuing it naively is unsafe.
• Stale model data. Schema changes while jobs are pending can break unserialization or model hydration. Deploy schema changes before code that depends on them, and drain queues during risky migrations.
• Redis as a single point of failure. If Redis restarts without persistence, pending jobs are lost. For workloads where job loss is unacceptable, evaluate SQS or a database driver with acceptable throughput, accepting the added cost or locking overhead.
• Very high concurrency. Beyond what a single Redis instance comfortably serves, move to SQS or a dedicated broker rather than scaling Redis vertically.

The design holds together because each piece addresses one failure class: Redis removes locking contention, bounded worker lifetimes remove memory decay, backoff absorbs transient faults, and Horizon surfaces everything else before users notice.