Airflow Scheduler HA: Metadata DB Schema Migration and Version Skew

Database Schema Migration in HA Environments

Apache Airflow 2.0+ supports high availability (HA) by running multiple schedulers simultaneously. While this architecture allows for rolling restarts of the scheduler components, the underlying metadata database migrations performed via airflow db upgrade are not transactional across the cluster.

Consistency Risks During Upgrades

When upgrading Airflow versions, there is a period where the database schema may be updated while some schedulers are still running the previous version of the application code. Because there is no native leader-election mechanism to coordinate schema migrations with scheduler lifecycles, this creates a potential version skew.

The uncertainty lies in how the scheduler handles a mid-migration schema state, specifically regarding task parsing and the stability of in-flight tasks that rely on XComs or specific trigger rules during the transition window.

• How does the scheduler behave when encountering a database schema that has been upgraded by a newer version of the code?
• Is it possible to maintain zero-downtime for task scheduling if mixed scheduler versions are temporarily active against the same metadata database?