Schema validation failure during Nomad job rollback after server upgrade

Question

Goal: Guarantee that a Nomad job can be rolled back to a previously stored specification without encountering schema validation errors after the Nomad server cluster has been upgraded to a newer version that introduced additional fields.

Constraint: The `nomad job rollback` command restores the exact job specification persisted in the Raft state, which may contain fields that are unknown to an older server version if the rollback target predates the upgrade. Nomad does not automatically downgrade or strip those unsupported fields, so the success of the rollback depends on the server’s ability to accept the spec as‑is.

Uncertainty: Whether Nomad should implement schema‑aware rollback that removes or transforms incompatible fields, or whether it should leave the responsibility to operators to pre‑validate or edit the spec before rolling back.

Questions: Should Nomad automatically strip unsupported fields during a rollback to an older server version? Is it sufficient to rely on operator‑driven pre‑validation and manual spec adjustment? What behavior would best balance safety and operational simplicity?

Wasi · Answer

Answer to the Core Question

No. Nomad's job rollback restores the persisted specification exactly as stored in Raft; it does not automatically strip or downgrade fields that are unknown to the server version that will receive the job. Operators must pre-validate or edit the spec before rolling back to an older cluster.

Likely Explanation

When a cluster is upgraded, the Raft state contains the latest job spec, which may include fields added in the newer API.
An older server will reject the spec during the job rollback because its JSON schema validator does not recognize those fields.
Automatically removing fields could silently drop custom metadata that other services rely on, and would hide configuration regressions.
Nomad's CLI offers no flag to downgrade a job spec; the only supported path is manual editing or external tooling.

Confirmed Facts (from current Nomad behavior)

The job rollback command does not modify the persisted spec.
Validation errors are surfaced immediately when the older server rejects the spec.
Operators can retrieve the persisted spec with nomad job status <job> -verbose and then edit it.
Validation can be tested locally with nomad job plan -validate <spec> on the older Nomad binary.
Running the edited spec on a test cluster that mirrors the older version is recommended before applying it to production.

Practical Steps for This Case

On the upgraded cluster, capture the persisted spec:
```
nomad job status my-job -verbose > my-job.json
```
Identify the fields that were added after the upgrade. These are usually marked with a version number or appear only in the newer spec.
- Example: upgrade_strategy, restart_policy, or consul stanza changes.
Remove or comment out unsupported fields. Keep required fields intact.
Validate locally against the older Nomad binary:
```
nomad job plan -validate my-job.json
```
If validation passes, spin up a temporary test cluster running the older Nomad version and apply the edited spec:
```
nomad job run my-job.json
```
Confirm the job starts and behaves as expected. If issues arise, iterate on the spec.
Once verified, perform the rollback on the production older cluster:
```
nomad job rollback my-job
```

Missing Diagnostic Detail Needed

To refine the recommendation, please provide the exact field(s) that the older server reports as unknown in the rollback attempt. This will help determine whether a simple removal is sufficient or if additional adjustments (e.g., schema versioning) are required.

Schema validation failure during Nomad job rollback after server upgrade

1 answer

Answer to the Core Question

Likely Explanation

Confirmed Facts (from current Nomad behavior)

Practical Steps for This Case

Missing Diagnostic Detail Needed

Your perspective belongs here.

0 question comments