ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Developer tools · #apacheairflow
A technical guide for choosing between Local, Celery, and Kubernetes executors in Apache Airflow based on scaling needs, infrastructure constraints, and fault tolerance.
Failed airflow db upgrade on 2.6.x When an Airflow upgrade applies Alembic migrations via airflow db upgrade and one script fails midway, the metadata database can be left in a partially migrated state. As of Airflow 2.6.x there is no built-in rollback command — no airflow db rollback exists, and Alembic's downgrade is not exposed through the Airflow CLI. Co
Apache Airflow 2.x utilizes a TaskInstance log handler that writes to a local directory by default. In ephemeral container environments, these local files are lost upon pod termination, necessitating a transition to remote logging providers such as AWS S3, Google Cloud Storage, or Azure Blob Storage. While setting remote_logging = True in airflow.cfg redirec
Apache Airflow 2.x provides a robust retry mechanism through the retries and retry_delay parameters, allowing failed task instances to be rescheduled automatically. However, the platform does not natively enforce idempotency for side effects occurring within operators, such as database insertions or API requests. While the TaskInstance object provides try_nu
In an Apache Airflow 2.x deployment, tasks executing on workers retrieve connection credentials from a secrets backend such as AWS Secrets Manager, Google Cloud Secret Manager, HashiCorp Vault, or environment variables. The goal is to enforce least-privilege access at the runtime boundary and handle expired credentials predictably before task dispatch. Web U
Data Consistency in Retry Cycles Apache Airflow manages task failures through a configurable retry mechanism. When a task fails during a write operation to an external database, a subsequent retry attempt may produce duplicate records if the initial attempt partially succeeded before the failure occurred. To maintain data integrity, tasks must be idempotent.
Goal: determine whether a repeatable local Docker Compose stack can faithfully reproduce production task-execution semantics when the executor is changed from CeleryExecutor to KubernetesExecutor. The official Airflow Docker Compose reference is architected around CeleryExecutor, with scheduler, worker, triggerer, Redis, and Postgres services. KubernetesExec
Determine whether Airflow’s stable REST API limit/offset pagination returns each dag run exactly once when new dag runs are created while a client pages through the collection. Constraints include the server‑imposed default and maximum page size, the absence of snapshot‑isolated ordering, version‑specific defaults for limit/offset, and the fact that the UI a
Handling Write Duplication During Task Retries Apache Airflow allows for automated task re-execution via the retries and retry_delay parameters. While these settings ensure task completion, they do not natively manage the atomicity of data writes to external sinks. A specific challenge occurs when a task fails after partially committing data or during a 'zom
kubernetes