Diagnosing and Resolving Zombie Tasks in Apache Airflow

Recognizable Condition

A zombie task in Apache Airflow is a task instance that the Scheduler believes is still running because its state in the metadata database is running, but the worker process that should be executing it has stopped sending heartbeats. The Scheduler periodically scans for such tasks and logs a warning: Detected zombie task. Downstream consequences include stuck DAG runs, blocked task slots, and inaccurate UI status.

Cause and Diagnostic Quick-Reference

Cause	Typical Symptom	Primary Verification
Worker OOM kill	Task disappears from worker logs; no failed state transition	Check dmesg -T | grep -i 'out of memory' on the worker node
Network partition / high latency	Heartbeat updates stop intermittently; task remains running in DB	Ping/latency tests between worker and metadata DB; check DB connection pool errors
Heartbeat timeout misconfiguration	Zombies appear under load even when workers are healthy	Review scheduler_heartbeat_sec and worker_heartbeat_sec in airflow.cfg
Worker process crash (segfault, SIGKILL)	No graceful shutdown; task instance heartbeat stops abruptly	Worker system logs (/var/log/messages, journalctl -u airflow-worker)

Ordered Diagnostic Checks

1. Confirm the zombie in the metadata database. Run the following query against the Airflow metadata DB (replace mydb with your schema):

   SELECT ti.task_id, ti.dag_id, ti.execution_date, ti.start_date, ti.last_heartbeat_at
   FROM task_instance ti
   WHERE ti.state = 'running'
     AND ti.last_heartbeat_at < NOW() - INTERVAL '5 minutes';
   

   Tasks returned with a stale last_heartbeat_at are candidates.
2. Check Scheduler logs for the detection message. On the Scheduler host, search:

   grep -i "Detected zombie task" /path/to/airflow/logs/scheduler/latest.log
   

   Note the task instance key (dag_id, task_id, execution_date) and timestamp.
3. Correlate with worker node system logs. On the worker that should have run the task (identifiable via hostname column in task_instance or from the log), examine:

   dmesg -T | grep -i -e 'out of memory' -e 'killed process'
   journalctl -u airflow-worker --since "1 hour ago" | grep -i -e error -e kill -e oom
   

   An OOM killer entry matching the task's PID confirms memory pressure.
4. Validate heartbeat configuration. Inspect airflow.cfg on both Scheduler and Worker:

   [scheduler]
   scheduler_heartbeat_sec = 5
   
   [core]
   worker_heartbeat_sec = 30
   

   If worker_heartbeat_sec is too low relative to GC pauses or DB latency, healthy workers may be marked as zombies.
5. Test network path to the metadata DB. From the worker, run:

   pg_isready -h $DB_HOST -p $DB_PORT -U $DB_USER
   # or for MySQL
   mysqladmin ping -h $DB_HOST -u $DB_USER -p
   

   Repeat during peak load; latency spikes > worker_heartbeat_sec cause missed heartbeats.

Fixes Tied to Findings

Worker OOM Kill

• Increase memory limits for the worker container/pod (e.g., Kubernetes resources.limits.memory).
• Profile the task's memory usage; consider chunking data or using XCom backend for large payloads.
• Enable task_heartbeat_rate_limit (Airflow 2.6+) to reduce heartbeat overhead.

Network Partition / Latency

• Co-locate workers and metadata DB in the same availability zone.
• Increase worker_heartbeat_sec to 60–120 seconds if latency is unavoidable.
• Configure DB connection pooling (sql_alchemy_pool_size, max_overflow) to absorb spikes.

Heartbeat Timeout Misconfiguration

• Set scheduler_heartbeat_sec to 5–10 seconds (default 5).
• Set worker_heartbeat_sec to at least 30 seconds; raise to 60+ under heavy load.
• Restart Scheduler and Workers after changes.

Worker Process Crash

• Upgrade Airflow and Python dependencies; segfaults often stem from C-extensions (e.g., pandas, numpy).
• Run workers with --autoscale or KubernetesExecutor to isolate task failures.
• Enable capture_task_logs and centralize logs for post-mortem.

Clearing a Confirmed Zombie (State Change – Rollback Possible)

If the task is genuinely dead (no process on any worker), you may manually mark it failed so the Scheduler can retry or mark downstream tasks appropriately.

-- Run in a transaction; verify row count first
BEGIN;
UPDATE task_instance
SET state = 'failed',
    end_date = NOW(),
    last_heartbeat_at = NOW()
WHERE dag_id = 'example_dag'
  AND task_id = 'example_task'
  AND execution_date = '2026-10-19 10:00:00'
  AND state = 'running';
COMMIT;

Rollback: If the task was actually still running (rare), the UPDATE will cause a duplicate execution when the worker eventually heartbeats. To roll back, restore the previous state from a DB backup or set state = 'running' and clear end_date immediately. Always verify no active process exists before updating.

Escalation Criteria

• Zombies recur daily after configuration changes and resource increases.
• Multiple workers report OOM kills simultaneously, indicating a platform-level memory issue.
• Network latency to the metadata DB exceeds 500 ms consistently.
• Scheduler logs show Detected zombie task for tasks that are confirmed running on workers (false positives).

Escalate to platform/infrastructure team with: Scheduler log snippets, DB query results, worker dmesg output, and current airflow.cfg heartbeat settings.

Limitations and Verification

• This guide assumes Airflow 2.2+ with a PostgreSQL or MySQL metadata database. Behavior may differ with SQLite or older versions.
• Heartbeat logic changes in Airflow 2.7+ (smart sensors, HA scheduler) – verify against your release notes.
• Manual DB updates bypass Airflow's ORM and signals; use only as last resort.

Practical verification: After applying fixes, trigger a test DAG with a long-running task. Simulate a worker kill (kill -9 on the task PID) and confirm the Scheduler marks the task failed within worker_heartbeat_sec + scheduler_heartbeat_sec seconds, without logging a zombie warning.