Question
Nova database schema compatibility boundary after failed rolling upgrade
Rey Maple
0 reputation · 08 Apr 2020, 06:58 UTC
120.5K views1
Loading…
0 reputation · 08 Apr 2020, 06:58 UTC
Goal is to define a safe operational boundary for Nova when a rolling upgrade fails after database migrations have been applied but before all compute and API nodes are updated.
The upgrade process relies on rolling updates for availability, and Nova requires schema version compatibility between API and database. Migrations can be additive or destructive, which creates tension between the desire for automated rollback and irreversible schema changes.
The permission boundary for starting services against a newer schema is not clearly documented for partial rollback scenarios. Clarity is needed on when a service start is considered safe versus when it must be blocked.
What constraint determines Nova API start eligibility against a newer database schema? Under what conditions is a partial rollback considered unsupported without restoring the database? Is there a documented configuration that enforces a schema version check before service start?
34775 reputation · 08 Apr 2020, 18:02 UTC
The eligibility of a Nova API service to start against a newer database schema is determined by the minimum required schema version defined in the service's code. If the database version is equal to or greater than the version the code expects, the service will typically start. However, this assumes the changes are additive (e.g., new tables or nullable columns).
A partial rollback—where API or compute nodes are reverted to a previous version while the database remains at the upgraded version—is considered unsupported under the following conditions:
In these scenarios, the system has crossed a hard compatibility boundary. The only supported recovery path is a full database restoration from a backup taken prior to the migration.
Nova does not typically use a separate configuration file to "toggle" schema checks; rather, the version check is baked into the database initialization logic. To verify the current boundary state, you can check the schema version directly in the database:
openstack# Example: Checking the schema version in the Nova database
# (Actual table names may vary by version; typically found in a versioning table)
SELECT * FROM nova.schema_version;SQLAlchemy or ProgrammingError exceptions.Required Diagnostic: To provide a definitive safety recommendation, please specify if any db version upgrade commands were executed that involved DROP or ALTER COLUMN (non-additive) operations.
Use comments to ask for clarification. Post a solution as an answer.
5,550 reputation · 08 Apr 2020, 15:14 UTC
Nova enforces a schema version gate at API startup: the code's minimum supported version is compared against the alembic_version table value. If the database schema is ahead of what the API binary expects, startup is blocked and logged as a version mismatch. This guard prevents compute nodes from operating against a newer schema that may lack backward‑compatible mappings. After a partial rolling upgrade, schema skew can appear even when individual nodes appear healthy. To verify the boundary, run nova-manage db version on a node and compare the output with the minimum schema version documented for the installed API release; a discrepancy will typically trigger a startup refusal in the logs. Operators should note that some deployment variants may relax this guard via variant‑specific settings, but doing so on an incompletely upgraded fleet risks data inconsistency from destructive migrations applied only on a subset of nodes.