Diagnosing Stuck Task Instances in Apache Airflow: A Field Guide for Engineering Teams

The Problem: Tasks Stuck in Queued State

When Airflow tasks remain stuck in queued state for extended periods—typically over 15 minutes—it signals a systemic issue rather than an isolated task failure. This condition can cascade, blocking downstream tasks and creating backlogs that are difficult to clear once established.

Recognizable Conditions

• Task instances appear in queued state in the UI despite no available workers
• airflow tasks list-runs --tree shows tasks with execution dates hours old still queued
• Worker logs show no pickup of tasks that have been queued for extended periods
• Database queries reveal task instances with start_date and end_date both NULL while state='queued'

Cause/Diagnostic Table

Symptom	Potential Cause	Diagnostic Check
Tasks queued >15 min with no worker pickup	Scheduler deadlock or database connection pool exhaustion	airflow jobs check and review scheduler logs for "Database connection" errors
Multiple workers unresponsive to new tasks	Worker process crash or container OOM kill	Check worker metrics endpoint: curl localhost:8793/metrics for restart counts
Redis/Broker connection failures in logs	Celery executor broker timeout under load	Verify Redis availability: redis-cli ping from worker nodes
Tasks fail with "NoneType object has no attribute"	Missing XCom values or upstream task failures	Query database: SELECT state, try_number FROM task_instance WHERE dag_id='' AND task_id='' ORDER BY execution_date DESC LIMIT 5

Ordered Diagnostic Checks

1. Verify scheduler health: Run airflow jobs check to confirm the scheduler process is responsive. If this hangs or fails, the scheduler may be deadlocked.
2. Check database connectivity: Execute airflow db check. Connection pool exhaustion manifests as "could not obtain db connection" errors in scheduler logs.
3. Inspect worker metrics: Query http://worker-host:8793/metrics (Prometheus endpoint) to check for container restarts, memory pressure, or CPU throttling that would prevent task execution.
4. Review broker availability: For Celery executor, test Redis connectivity from worker nodes using redis-cli -h ping. Broker unavailability causes tasks to remain queued indefinitely.
5. Query task instance states directly: Connect to the Airflow metadata database and run:

   SELECT dag_id, task_id, state, queued_dttm, start_date
   FROM task_instance 
   WHERE state='queued' 
   AND queued_dttm < NOW() - INTERVAL '15 minutes'
   ORDER BY queued_dttm ASC
   LIMIT 10;

   This identifies exactly which tasks are stuck and for how long.

Fixes Tied to Findings

Scheduler deadlock: Restart the scheduler process. In Kubernetes environments, this typically resolves the issue within 2-3 minutes as the new scheduler instance processes the backlog.

Database connection pool exhaustion: Increase pool size in airflow.cfg ([database] section):
sql_alchemy_pool_size = 30
sql_alchemy_pool_overflow = 10
Then restart the webserver and scheduler.

Worker OOM kills: Identify the problematic task by examining worker container logs for exit code 137. Common causes include unbounded data processing in PythonOperators or loading large datasets into memory. Fix by adding chunked processing or increasing worker memory limits.

Broker unavailability: Verify Redis persistence settings and memory limits. For high-throughput workloads, consider increasing Redis maxclients and enabling swapfree alerts.

Missing XCom dependencies: Tasks waiting for upstream results that were never produced will remain queued. Check upstream task logs for failures, then either fix the root cause or implement proper error handling to prevent silent failures.

Escalation Criteria

• If tasks remain queued after scheduler restart for more than 10 minutes, escalate to database team—potential deadlock in task_instance table
• If worker metrics show persistent memory pressure (>90% usage) across all workers, the executor configuration may need scaling
• If broker connection failures persist under normal load (<50% cluster capacity), infrastructure issues require escalation

Verification Steps

After applying fixes, verify resolution using:

1. airflow tasks list-runs --tree <dag_id> to confirm tasks are progressing through states
2. Monitor the metrics endpoint for 5-10 minutes to ensure no new stuck tasks appear
3. Query the database to confirm state='queued' count returns to normal baseline

Limitations and Considerations

In Airflow 2.x, task instance mapping changes failure propagation behavior—mapped task groups can create thousands of individual task instances that compound queue issues. Always check for dynamic task generation when diagnosing stuck tasks.

XCom size limits (default 1MB) can cause unexpected serialization failures when tasks attempt to pass large payloads. Monitor XCom size in logs and consider external storage mechanisms for large data transfers.