Question
Alembic Downgrade vs. Metadata Database Restore for Failed Upgrades
Kavi Vale
0 reputation · 21 Feb 2023, 13:39 UTC
123.3K views1
Loading…
0 reputation · 21 Feb 2023, 13:39 UTC
When performing an Apache Airflow upgrade, the airflow db upgrade command applies incremental schema revisions via Alembic. If a migration fails mid-process, the metadata database may be left in an inconsistent state, necessitating a recovery action before the scheduler or webserver can be safely restarted.
There are two primary documented paths for reverting a failed migration, each presenting different trade-offs regarding data integrity and operational continuity:
airflow db downgrade <revision> to revert specific schema changes. This is faster but may leave orphaned tables or partial data inserts that the migration script did not explicitly handle in its downgrade logic.Given that Airflow lacks an automated rollback mechanism for failed migrations, the choice depends on whether schema purity or the preservation of recent task metadata is the priority.
A downgrade cannot return the metadata database to a state compatible with the previous Airflow version when the migration script lacks a complete downgrade() function, when data transformations were applied without reversal logic, or when the failure occurred during non-transactional DDL. Databases such as MySQL do not support transactional DDL for all statements, so a partial upgrade can leave tables, columns, or indexes in an indeterminate state that airflow db downgrade cannot automatically repair.
There is no single documented command that guarantees logical schema equivalence after a manual downgrade. You can run scoped checks, but they confirm accessibility, not compatibility.
Likely explanation: airflow db upgrade applies Alembic revisions sequentially. If the process stops mid-way, alembic_version may reference a revision whose upgrade() only partially executed.
Behaviors requiring version-specific verification:
apacheairflowdowngrade().SELECT version_num FROM alembic_version;downgrade() and the database supports transactional DDL, attempt:airflow db downgrade <revision>airflow db checkWhich metadata database engine are you using? This changes the recommendation because transactional DDL support differs between PostgreSQL and MySQL.
This guidance assumes Apache Airflow 2.x with Alembic-managed migrations. Revision hashes, downgrade script completeness, and database behavior vary by Airflow version. Because current verification is required, treat this as a draft for human review before executing destructive commands in production.
Use comments to ask for clarification. Post a solution as an answer.
5,550 reputation · 21 Feb 2023, 20:41 UTC
After a manual airflow db downgrade you can bring the alembic_version table in line with the intended revision by stamping it directly:
alembic stamp
This tells Alembic that the database is now at that revision, assuming you have manually corrected any schema objects that the downgrade script did not remove.
To verify that the schema matches what the target Airflow version expects, compare the current schema with a clean schema generated for that version. For PostgreSQL you can dump the schema only:
pg_dump --schema-only --no-owner --no-acl airflow_db > current.sql
pg_dump --schema-only --no-owner --no-acl airflow_db_target > target.sql
diff -u target.sql current.sql
For MySQL/MariaDB the equivalent is mysqldump --no-data --skip-comments. Any remaining differences indicate objects that were not fully reverted and may cause compatibility issues.
Finally, run airflow db check (available from Airflow 2.0) to confirm that the database can be used by the scheduler and webserver; it will report missing tables, columns, or indexes but does not validate data content.