ReadMeFeed / Community knowledge

#apache-airflow

Real questions. Useful conversations. Find the people who know your stack.

Article0 replies4,325 followingPosted 14 Aug 2025, 21:53 UTC56.9K views3,079 votes

Architecting Data Flow in Airflow: Transitioning to the TaskFlow API

Learn how to use the Apache Airflow TaskFlow API to eliminate XCom boilerplate, manage data boundaries, and avoid metadata database bloat in your pipelines.

Community & Careers · Guide

apache-airflow

apacheairflow taskflow-api

Tasadduq Burneyowner

Started14 Aug 2025, 21:53 UTC

Level 38Owner35,775 rep points

Community & Careers

Article0 replies2,462 followingPosted 24 May 2026, 23:49 UTC99.3K views2,182 votes

Choosing the Right Airflow Executor for Your Workload

A decision guide for selecting an Apache Airflow executor (Local, Celery, Kubernetes, CeleryKubernetes) based on workload scale, infrastructure constraints, and operational trade-offs, with configuration examples and validation steps.

Community & Careers · Guide

kubernetes-executor apache-airflow

apacheairflow

Community & Careers

Question0 answers2,331 followingPosted 03 May 2022, 20:23 UTC83.2K views4,805 votes

Can the Airflow scheduler be stopped between runs with LocalExecutor without losing scheduled tasks?

Goal Reduce operating cost for a low‑traffic Apache Airflow deployment by possibly stopping the scheduler node between DAG runs while using LocalExecutor. Constraints and uncertainties The decision depends on the catchup setting, whether any DAGs use data‑aware timetables, the acceptable latency for triggering new runs, and the impact on DAG parsing, health‑

Community & Careers · Editorial question

localexecutor scheduler apache-airflow

Community & Careers

Question0 answers3,941 followingPosted 05 Mar 2022, 00:26 UTC123.6K views749 votes

LocalExecutor vs CeleryExecutor for filesystem-dependent tasks

When transitioning from a local development environment to a production cluster, DAGs that rely on shared local filesystems often fail. The LocalExecutor executes tasks as subprocesses on the scheduler node, allowing tasks to read and write files from the same /tmp directory seamlessly. However, moving to CeleryExecutor distributes tasks across remote worker

Community & Careers · Editorial question

localexecutor filesystem apache-airflow