ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Tasks stuck in queued state for over 15 minutes indicate scheduler deadlock or database connection pool exhaustion. Follow this diagnostic guide with ordered checks and database queries to identify and resolve the root cause.
A step-by-step diagnostic guide for Apache Airflow zombie tasks: identify the mismatch between scheduler and worker state, pinpoint root causes (OOM, network, config), run targeted checks, apply fixes, and know when to escalate.
Scale your Airflow pipelines by generating DAGs in a single Python file using a for‑loop. Learn how to keep the scheduler healthy, ensure unique dag_ids, and decide when to use dynamic task mapping instead.
Enable Airflow’s DAG serialization to cut scheduler and UI load times. Learn how it stores DAGs in JSON, verify it in the database, and avoid common pitfalls like non‑serializable objects and database bloat.
Dynamic DAG generation lets you create dozens of Airflow pipelines from a single Python file, cutting boilerplate and simplifying maintenance. Learn how to build a factory, avoid scheduler pitfalls, and keep your Web UI responsive.
An architecture note on using Apache Airflow deferrable operators to release worker slots during I/O waits, covering trust boundaries, monitoring, and failure modes.
Static Airflow DAGs make data-dependent fan-out awkward. Dynamic Task Mapping expands tasks at runtime from upstream XCom values, but it changes visibility, retries, and scheduler load.
Deferrable operators let Airflow sensors yield control instead of holding worker slots. Here is how the Triggerer changes concurrency math, a worked file-sensor comparison, and what to verify before production.
Apache Airflow's TaskFlow API replaces operator boilerplate with plain Python functions, automating XCom handling and enabling dynamic task mapping via .expand(). This blog shows a concrete DAG example, explains executor-agnostic behavior, and highlights metadata-load limits.
apacheairflow
python