Question
Nomad auto_revert behavior during external schema migrations
Sky Maple
0 reputation · 25 Feb 2022, 02:28 UTC
53.7K views1
Loading…
0 reputation · 25 Feb 2022, 02:28 UTC
HashiCorp Nomad provides the update stanza to manage rolling updates, utilizing auto_revert = true to automatically restore a previous job version if health checks fail. This mechanism ensures that the application binary and configuration return to a known stable state.
A challenge arises when a deployment includes an external database schema migration. While Nomad can revert the task group to a previous version, it lacks native integration to trigger a corresponding rollback of the external data schema. This creates a potential mismatch where a reverted application version attempts to interact with a migrated schema that is no longer compatible.
Given the current lifecycle management in Nomad 1.x, how can the auto_revert process be coordinated with external state changes to prevent version mismatch? Is there a documented method to trigger external cleanup scripts specifically upon a revert event?
34775 reputation · 25 Feb 2022, 04:43 UTC
Nomad’s auto_revert = true only rolls back the job specification and task images; it does not provide a hook that runs exclusively on a revert event to trigger external schema rollback scripts.
nomadupdate stanza with auto_revert reverts the allocation to a previous job version when health checks fail.prestart, poststart, poststop) execute on every task start or stop, regardless of whether the job is being rolled forward or backward.To keep the application binary and the external database schema in sync, you must make the schema version check part of the task’s startup logic. The task can compare the version of the running job (available via Nomad‑provided environment variables) with a stored schema version and run a downgrade script when they diverge.
prestart hook (or an init container/sidecar) that:NOMAD_JOB_NAME and NOMAD_JOB_VERSION from the environment.Do you already have a centralized version store (such as Consul KV or a database table) that the task can read? If not, implementing one will change the recommendation, as the version‑comparison step relies on an external source of truth.
Use comments to ask for clarification. Post a solution as an answer.
5,940 reputation · 25 Feb 2022, 11:54 UTC
While the previous response suggests managing version checks within the application logic, it is worth noting that Nomad's lifecycle hooks can be leveraged to handle cleanup during the revert process. Specifically, the poststop hook of the failing task group will execute as Nomad tears down the unsuccessful deployment to make room for the reverted version.
If you implement rollback logic in a poststop hook, keep these operational constraints in mind for Nomad 1.x:
poststop hook of the failed version runs before the prestart hook of the reverted stable version.nomad alloc logs -stderr <allocation-id> to ensure the cleanup script triggered as expected.