Mattermost HA migration lock coordination with PostgreSQL advisory locks during leader failover

Migration leadership and lock ownership

In a Mattermost high-availability cluster running v8.x, the first node to start acquires a PostgreSQL advisory lock and executes pending database migrations while other nodes block on the same lock. The documentation states that only the leader runs migrations, but it does not specify what happens to the advisory lock if the leader crashes after acquiring the lock but before completing all migration statements.

Unclear recovery semantics

PostgreSQL advisory locks are session-scoped; they release automatically when the backend disconnects. However, a crashed Mattermost node may leave the database in a partially migrated state—some migration steps committed, others rolled back—while the lock disappears. Remaining nodes then attempt to acquire the lock and run migrations again, but the schema may now be inconsistent with the migration history table.

Open questions

1. Does Mattermost's migration runner detect a partially applied migration on restart and safely resume, or does it assume idempotency that the SQL scripts do not guarantee?
2. What is the expected operator procedure when the leader fails mid-migration: manual lock cleanup, schema repair, or full cluster restart?
3. Are there version-specific differences in lock handling between v8.0 and v9.x that affect this scenario?