Airflow Deferrable Operators: Moving Sensor Wait Time Off Your Workers

The sensor that ate your worker slots

You have a DAG that waits for a file to appear in cloud storage. The sensor polls every few minutes, and while it waits, it holds a worker slot. If you run dozens of these sensors, your workers fill up with tasks that are essentially sleeping. You can raise poke_interval or switch to mode="reschedule", but the core issue remains: waiting is still consuming scheduling capacity.

Airflow's deferrable operators offer a different trade. Instead of holding a worker slot while polling, the task yields control back to Airflow and lets a dedicated component—the Triggerer—do the waiting. This article walks through the mechanism, a concrete file-sensor comparison, and the operational costs you inherit when you adopt it.

What blocking sensors actually cost

A traditional sensor runs inside a worker process. In poke mode, it occupies a slot for the entire polling duration. In reschedule mode, it frees the slot between pokes, but the task still wakes up, consumes scheduler and worker resources, and repeats. For long waits—minutes or hours—the overhead adds up, and your effective concurrency is lower than your raw worker count suggests.

Deferrable operators change this. When a task defers, it enters a special state and releases its worker slot entirely. The polling logic moves into an asynchronous trigger that runs in a separate process. Your worker slot is free for other tasks, and the scheduler is not repeatedly waking the same task.

The Triggerer: a dedicated asyncio process

Introduced in the Airflow 2.x line, the Triggerer is a standalone component that runs trigger callbacks inside a single asyncio event loop. When a deferrable operator defers, it registers a trigger instance with the Triggerer. That trigger periodically checks the external condition—without blocking a worker. When the condition is met, the trigger signals Airflow, and the task resumes on the next available worker.

Because the event loop is single-threaded, every trigger must be written as async, non-blocking code. A slow HTTP call or a CPU-bound computation inside a trigger will stall unrelated deferred tasks. This is the most important constraint to remember when writing custom triggers.

Worked example: waiting for a file in cloud storage

Suppose you need to wait for data-2026-10-15.csv to land in a cloud storage bucket. The exact class names and parameter support vary across Airflow 2.x releases and provider packages, so verify against the documentation bundled with your deployment. The pattern below illustrates the shape of the migration.

Blocking version

# Replace with your provider's actual sensor class.
# Verify parameter names against your Airflow 2.x docs.
wait_for_file = CloudStorageSensor(
    task_id="wait_for_file",
    bucket_name="my-bucket",
    object_name="data-2026-10-15.csv",
    poke_interval=120,
    timeout=3600,
    mode="reschedule",
)

Deferrable version

# Many sensors in recent Airflow 2.x provider releases accept
# a deferrable parameter. Confirm availability for your version.
wait_for_file = CloudStorageSensor(
    task_id="wait_for_file",
    bucket_name="my-bucket",
    object_name="data-2026-10-15.csv",
    deferrable=True,
)

With deferrable=True, the task instance enters a deferred state. The Triggerer assumes responsibility for polling. If the Triggerer is healthy, the worker slot is released immediately. When the file appears, the task resumes and executes the rest of the DAG.

If your sensor does not yet support deferral, the alternative is writing a custom trigger class that implements an asynchronous run method. This gives you the same worker-slot savings, but it moves you from simple Python callables into asyncio territory, where testing and debugging require more deliberate effort.

Trade-offs you inherit

Deferral is not free. You are adding a stateful, single-threaded dependency to your Airflow deployment.

• Triggerer availability becomes critical. If the Triggerer stops, deferred tasks do not resume. You need to monitor it as carefully as you monitor the scheduler.
• Asyncio is unforgiving. A trigger that makes a synchronous HTTP call or performs heavy computation will block the event loop and stall every other deferred task in the same Triggerer.
• Debugging surface grows. A failing synchronous sensor produces a clear Python traceback. A failing async trigger requires reasoning about event-loop state, and the failure may surface in Triggerer logs rather than task logs.
• Semantics can shift. Timeout, retry, and SLA behavior may differ subtly when a task is deferred versus running synchronously. Do not assume identical behavior without testing.

Before you roll it out

Deferrable operators are a capacity optimization, not a default. Adopt them where sensors wait for external systems for minutes or longer, and where worker slots are genuinely scarce.

1. Check the Airflow documentation bundled with your deployment to confirm which sensors support deferrable=True in your version.
2. Run airflow triggerer locally or in a staging environment—typically as the same user that runs your Airflow services—and deploy a small DAG with both a blocking and a deferrable sensor.
3. Inspect worker slot usage and Triggerer logs in the Airflow UI. Confirm that the deferrable task releases its slot and that the Triggerer resumes it cleanly.
4. Only then promote the pattern to production DAGs, and add Triggerer health to your monitoring stack.

If your sensors are short-lived or your workers are underutilized, the added operational complexity may not pay off. But when wait time dominates your DAG runtime, deferral can turn idle workers into useful ones.