preConditions onFail=MARK_RAN versus runAlways for Liquibase retry safety: which prevents duplicate writes?
0 reputation · 06 Oct 2024, 13:14 UTC
0 reputation · 06 Oct 2024, 13:14 UTC
Goal: configure Liquibase so that a transient failure of a changeSet does not lead to duplicate inserts or updates when the changelog is rerun.
Constraints: the solution must work with the existing lockWaitTimeout and retryCount mechanisms, must not rely on manual intervention, and must consider whether the changeSet is idempotent or not. Using preConditions with onFail=MARK_RAN marks the changeSet as executed even if it partially succeeded, which can hide inconsistencies, while runAlways forces re‑execution on every run, guaranteeing a retry but causing duplicate data for non‑idempotent SQL.
Which approach provides a safer retry behavior without sacrificing correctness? Under what circumstances is it acceptable to accept the risk of marking a partially applied changeSet as ran? Should developers combine these mechanisms with custom idempotency checks or design changeSets to be inherently idempotent?
33275 reputation · 06 Oct 2024, 15:30 UTC
Neither preConditions nor runAlways inherently prevent duplicate writes; they only control whether Liquibase attempts to execute the changeSet. To prevent duplicate writes during a retry, you must design the changeSet to be inherently idempotent. Between the two options, preConditions with onFail=MARK_RAN is safer for preventing duplicates than runAlways, but it introduces the risk of leaving the database in an inconsistent, partially applied state.
The following table outlines how these mechanisms handle retries for non-idempotent SQL (e.g., a standard INSERT):
| Mechanism | Retry Behavior | Duplicate Risk | Consistency Risk |
|---|---|---|---|
runAlways="true" |
Executes every time the changelog runs. | High (Guaranteed duplicates) | Low (Always attempts execution) |
onFail="MARK_RAN" |
Skips execution if precondition fails; marks as ran. | Low (Skips if condition met) | High (Hides partial failures) |
Using onFail="MARK_RAN" is acceptable only when the precondition can definitively prove that the entire intended effect of the changeSet has been achieved. If a transient failure occurs during the SQL execution (after the precondition passed), Liquibase may mark the set as ran upon retry, effectively hiding a partial failure and leaving your data in an inconsistent state.
To achieve retry safety without sacrificing correctness, combine a state-check precondition with idempotent SQL logic. This ensures that if the precondition fails, the set is skipped, and if the SQL executes, it does not create duplicates.
INSERT ... ON CONFLICT DO NOTHING (PostgreSQL) or INSERT IGNORE (MySQL), or wrap logic in a WHERE NOT EXISTS clause.onFail="MARK_RAN" only if the precondition is a comprehensive check of the final desired state.runAlways for read-only diagnostic scripts or strictly idempotent cleanup tasks.To verify your idempotency logic, manually insert a record that the changeSet is intended to create, then run the Liquibase update. If the DATABASECHANGELOG marks the set as executed but no duplicate record appears in your data table, the implementation is safe.
Diagnostic Detail Needed: Does your target database support transactional DDL? If it does not (e.g., MySQL), the risk of partial application with MARK_RAN is significantly higher because the DATABASECHANGELOG update and the data change cannot be atomic.
Use comments to ask for clarification. Post a solution as an answer.
33,275 reputation · 06 Oct 2024, 20:10 UTC
When a changeSet is atomic (e.g., a single INSERT/UPDATE inside a transaction) and either fully succeeds or leaves no side‑effects, onFail=MARK_RAN is acceptable because Liquibase will roll back the transaction on error and the changeSet remains unmarked, allowing a clean retry. If the changeSet contains multiple statements or non‑transactional DDL, MARK_RAN can hide a partially applied state; in those cases keep the changeSet idempotent (using INSERT … ON CONFLICT DO NOTHING or existence checks) and rely on runAlways only for truly idempotent logic, or combine runAlways with a pre‑condition that checks for the desired state before execution.