Airflow Scheduler HA: Metadata DB Schema Migration and Version Skew

Answer: Scheduler behavior and zero-downtime feasibility

When airflow db upgrade runs, the metadata database schema is updated globally while any scheduler still running the older version continues to use its SQLAlchemy models. If the newer schema does not match those models (e.g., a column was renamed, dropped, or a constraint added), the older scheduler will raise sqlalchemy.exc.ProgrammingError or OperationalError during DAG parsing or the scheduling loop, often causing the scheduler to crash or enter a crash-loop. This is the confirmed fact: mixed versions lead to ORM mismatches and scheduler failure.

Likely explanation

The scheduler has no leader‑election or migration‑coordination mechanism; each instance independently connects to the same database. Therefore, during the window between the migration start and the restart of all schedulers on the new code, the schema seen by old schedulers is newer than their code expects, producing the errors described above.

Steps for this case

1. Stop all scheduler and webserver instances to ensure no active connections are using the old models.
2. Run airflow db upgrade from a single admin node.
3. Verify the migration completed with airflow db check-migrations (or by checking the alembic_version table).
4. Start one scheduler instance on the new code and confirm that DAG parsing succeeds and the scheduler logs show no SQLAlchemy errors.
5. Scale back to the desired number of scheduler instances.

Missing diagnostic detail: Please specify the exact Airflow version jump (e.g., from 2.5.1 to 2.8.0) so we can assess whether the migration is additive or contains breaking changes that would affect the risk assessment.