Drupal Queue API Architecture Note: Reliable Background Jobs Without a Broker

The request-lifecycle problem

A Drupal page request that spends thirty seconds importing a feed or sending five hundred emails will hit PHP timeouts, exhaust worker processes, and leave users staring at a spinner. When the request dies mid-operation, the work is half-finished and there is no automatic retry. The Queue API moves that work out of the request lifecycle into a durable table that a separate consumer processes, so the HTTP response returns immediately and background work can be retried until it succeeds.

Requirements for a background boundary

• Decouple execution from the HTTP response so slow work does not block users.
• Provide retry semantics for transient failures such as a remote API timeout.
• Keep payloads small, serializable, and inspectable without executing arbitrary code.
• Survive consumer crashes: an item must remain available until it is explicitly acknowledged.
• Offer operational visibility into queue depth, latency, and error rates.

The smallest suitable design

In Drupal 8 and later, the idiomatic way to define a queue is a QueueWorker plugin. The worker declares which queue it consumes and implements processItem(). Drupal's cron system invokes workers automatically, so the smallest viable design is a single plugin plus a call to createItem() from trusted code.

<?php

namespace Drupal\acme_notify\Plugin\QueueWorker;

use Drupal\Core\Annotation\QueueWorker;
use Drupal\Core\Queue\QueueWorkerBase;

/**
 * Processes notification emails.
 *
 * @QueueWorker(
 *   id = "acme_notify_email",
 *   title = @Translation("Acme notification email"),
 *   cron = {"time" = 60}
 * )
 */
class AcmeNotifyQueueWorker extends QueueWorkerBase {

  /**
   * {@inheritdoc}
   */
  public function processItem($data) {
    // Validate payload before use.
    if (!is_array($data) || !isset($data['uid'], $data['subject'], $data['body'])) {
      throw new \InvalidArgumentException('Malformed queue payload.');
    }
    $uid = (int) $data['uid'];
    $subject = (string) $data['subject'];
    $body = (string) $data['body'];
    // Send mail or call an external service here.
  }

}

Push work from a form submit, hook, or Drush command:

<?php

$queue = \Drupal::service('queue')->get('acme_notify_email');
$queue->createItem([
  'uid' => 123,
  'subject' => 'Your report is ready',
  'body' => 'The monthly report has been generated.',
]);

Drupal 7 sites used hook_queue_info() to declare queues; that hook no longer exists in modern Drupal, so new code should use the plugin system. If you need a queue that is not tied to a fixed worker, you can still retrieve any named queue with \Drupal::queue('machine_name') and process it manually.

Trust and data boundaries

Queue items are serialized before they are written to the database. The default database queue serializes the payload with PHP's serialize(). That creates two boundaries you must respect:

1. Producers must be trusted. Only application code should call createItem(). Never accept queue names or payloads directly from untrusted user input, because a crafted payload can become an object-injection vector when it is unserialized.
2. Consumers must validate. Treat every item as potentially malformed. Cast expected scalars, check required keys, and reject anything unexpected before passing it to mail, database, or filesystem APIs.

Avoid storing full entity objects in the queue. Entity schemas and class definitions change across Drupal upgrades, and a serialized object from an older version may fail to unserialize or behave unexpectedly. Store identifiers and simple scalars instead, then load the entity fresh inside the worker.

Operational checks

Background work fails silently unless you measure it. Use the following checks in staging and production.

Check	How to observe	Expected signal	Risk if ignored
Queue depth	drush queue:list or the queue database table	Count is stable or trending down	Back-pressure; producer outpaces consumer
Processing latency	Custom log entry or watchdog timestamp when item is created vs. processed	Within your SLA (e.g., under five minutes)	Stale notifications; user-visible delays
Error rate	Watchdog / dblog entries from the worker's logger channel	Low and non-recurring	Poison-pill items blocking the queue
Cron health	External cron monitor or drush cron exit status	Recent successful run	Workers never execute; queue stalls

Command notes: Run drush queue:list from the Drupal webroot or via a site alias; it requires shell access and a Drush version compatible with your site. The expected result is a table of queue names with item counts. To process a queue manually, run drush queue:run acme_notify_email. Be aware that a large backlog can exceed the PHP CLI time limit, so prefer cron-driven batches or increase max_execution_time for that command.

Failure modes and mitigations

Stalled cron

Drupal's Queue API relies on cron to trigger workers. If cron stops running—because the server cron entry was removed, the cron key changed, or a fatal error aborts the run—items sit in the table untouched. Mitigate this with an external cron monitor that alerts when cron.php or drush cron has not executed within the expected interval.

Poison-pill items

An item that always throws an exception will never be deleted. On the next cron run the worker claims it again, fails again, and the queue makes no progress. The Queue API does not provide a built-in retry limit or dead-letter queue, so you must implement one. A simple pattern is to catch the exception, increment an attempt counter in the payload, and re-queue the item up to a threshold; after that, move it to a separate dead-letter queue for manual inspection.

<?php

public function processItem($data) {
  $attempts = $data['attempts'] ?? 0;
  try {
    // Perform work.
  }
  catch (\Exception $e) {
    if ($attempts >= 3) {
      \Drupal::queue('acme_notify_email_dead')->createItem($data);
      \Drupal::logger('acme_notify')->error('Dead-lettered item after @n attempts.', ['@n' => $attempts]);
      return;
    }
    $data['attempts'] = $attempts + 1;
    \Drupal::queue('acme_notify_email')->createItem($data);
    throw $e;
  }
}

Note that re-queueing inside processItem() creates a new row; the original item is still considered unprocessed and will be retried by cron unless you delete it. For a true dead-letter path, delete the original item after copying it to the dead-letter queue.

Back-pressure

When producers add items faster than a single cron worker can process them, the queue grows. Batch processing helps, but the database queue is not designed for unbounded concurrency. If depth grows monotonically under normal load, the design has outgrown the default implementation.

Conditions that change the design

• Throughput above roughly 10,000 items per minute. A single database-backed queue with cron-driven workers becomes a bottleneck. Move to a dedicated message broker such as RabbitMQ or Redis using a contrib queue plugin, and run multiple worker processes.
• Regulatory data-residency or encryption requirements. Payloads in the default queue table are serialized PHP in the primary database. If regulations require encrypted queue payloads or a separate data store, an external broker or an encrypted queue implementation is necessary.
• Sub-minute latency requirements. Cron granularity is typically minutes. If jobs must start within seconds, use a dedicated queue runner triggered by a system daemon or an external scheduler rather than Drupal cron.
• Exactly-once semantics. The Queue API guarantees at-least-once delivery. If duplicate side effects are unacceptable, make consumers idempotent by checking a processed-items table or using an external transactional outbox.

Practical verification

1. Enqueue a test item with a known payload, for example a log message that includes a unique token.
2. Trigger processing with drush queue:run acme_notify_email or by running cron.
3. Confirm the expected side effect occurred exactly once: the unique token appears in the log, or the test email is delivered.
4. Inspect watchdog entries for the worker's logger channel and verify there are no unexpected errors.
5. Check the queue table or drush queue:list to confirm the item was removed and the queue is empty.

Limitations

The Queue API is a durable job table, not a real-time streaming platform. It depends on cron or an external trigger, offers no native dead-letter mechanism, and serializes payloads in a way that can break across major Drupal upgrades. For most sites it is the smallest suitable design for decoupling slow work from HTTP requests; when volume, latency, or compliance requirements exceed those boundaries, an external broker is the appropriate next step.