Selecting the Right Apache Airflow Executor for Your Workload
A technical guide for choosing between Local, Celery, and Kubernetes executors in Apache Airflow based on scaling needs, infrastructure constraints, and fault tolerance.
Loading…
ReadMeFeed / Community knowledge
#apache-airflow
Real questions. Useful conversations. Find the people who know your stack.
kubernetesRemote Logging Configuration Transition for Ephemeral Worker Environments
Apache Airflow 2.x utilizes a TaskInstance log handler that writes to a local directory by default. In ephemeral container environments, these local files are lost upon pod termination, necessitating a transition to remote logging providers such as AWS S3, Google Cloud Storage, or Azure Blob Storage. While setting remote_logging = True in airflow.cfg redirec
apacheairflowDuplicate or missing entries when paginating Airflow REST API with limit/offset
Determine whether Airflow’s stable REST API limit/offset pagination returns each dag run exactly once when new dag runs are created while a client pages through the collection. Constraints include the server‑imposed default and maximum page size, the absence of snapshot‑isolated ordering, version‑specific defaults for limit/offset, and the fact that the UI a
Airflow Task Retry Limits and Data Idempotency
Handling Write Duplication During Task Retries Apache Airflow allows for automated task re-execution via the retries and retry_delay parameters. While these settings ensure task completion, they do not natively manage the atomicity of data writes to external sinks. A specific challenge occurs when a task fails after partially committing data or during a 'zom
Apache Airflow and SQLAlchemy: Connection Pool Exhaustion in Distributed Executor Environments
Apache Airflow relies on SQLAlchemy to manage the connection pool for its metadata database. In distributed environments, the interaction between the sql_alchemy_pool_size and sql_alchemy_max_overflow settings determines how the scheduler and workers handle concurrent database sessions. When scaling worker nodes via the Celery or Kubernetes executors, there
sqlalchemy