ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Schema Recovery Strategies When performing an Apache Airflow upgrade, the airflow db upgrade command applies incremental schema revisions via Alembic. If a migration fails mid-process, the metadata database may be left in an inconsistent state, necessitating a recovery action before the scheduler or webserver can be safely restarted. Recovery Constraints The
Goal: Determine which alerting mechanism best reduces notification noise for SLA misses and task failures while maintaining scheduler stability. Built‑in email alerts are enabled by default, can cause repeated emails during retries and backfills; callback‑driven external alerts require scheduler‑side callbacks that may delay scheduling if external API latenc
We run Apache Airflow 2.x on CPython and are planning upgrades across the Airflow↔Python boundary. Each Airflow minor release declares its supported Python range via the Requires-Python package metadata, and Airflow publishes per-Python constraints files for installation, but there seems to be no published schedule for when an end-of-life Python version is d
Goal Reduce operating cost for a low‑traffic Apache Airflow deployment by possibly stopping the scheduler node between DAG runs while using LocalExecutor. Constraints and uncertainties The decision depends on the catchup setting, whether any DAGs use data‑aware timetables, the acceptable latency for triggering new runs, and the impact on DAG parsing, health‑
When enforcing least privilege in Airflow, we need active web sessions and worker-resolved secrets to reflect credential changes promptly—for example, after rotating a connection password in an external secrets backend or disabling a user account. The goal is to understand the documented boundaries of this behavior so we can decide whether Airflow alone can
Apache Airflow ships airflow db backup and airflow db restore to export and reimport the metadata database (PostgreSQL, MySQL, or SQLite) for disaster recovery. The restore command finishes without error, but there is no documented validation step that compares the restored database against the backup file itself — no row-count or checksum comparison is buil
When transitioning from a local development environment to a production cluster, DAGs that rely on shared local filesystems often fail. The LocalExecutor executes tasks as subprocesses on the scheduler node, allowing tasks to read and write files from the same /tmp directory seamlessly. However, moving to CeleryExecutor distributes tasks across remote worker
Apache Airflow is transitioning toward a decoupled execution model via the Task SDK to reduce memory overhead by separating task execution logic from the core environment. This architectural shift aims to mitigate resource leaks associated with loading the full Airflow core in every worker process. A specific uncertainty exists regarding the state management
Database Schema Migration in HA Environments Apache Airflow 2.0+ supports high availability (HA) by running multiple schedulers simultaneously. While this architecture allows for rolling restarts of the scheduler components, the underlying metadata database migrations performed via airflow db upgrade are not transactional across the cluster. Consistency Risk
apacheairflow