Question
Event Upcasters vs. Versioned Event Types for Rollback-Safe Schema Evolution
Rahul Cedar
0 reputation · 08 Dec 2023, 06:29 UTC
25.4K views
Loading…
0 reputation · 08 Dec 2023, 06:29 UTC
When updating the event schema for a Persistent Actor, the primary goal is ensuring that a previous binary version can still recover state if a deployment must be rolled back after new events have been written to the journal.
Akka Persistence provides EventAdapter and upcaster chains to handle forward compatibility, allowing newer code to interpret legacy events. However, Akka does not provide a native "downcasting" mechanism. If a new binary writes events with added fields or changed structures, an older binary may fail to deserialize these events during replay, leading to actor recovery failure.
Two common patterns exist to mitigate this risk:
UserUpdatedV2) and maintaining separate reader logic for both versions during the transition period.While additive changes simplify the event stream, snapshots created by the new binary may remain opaque binary blobs that are incompatible with the old binary, potentially bypassing event-level compatibility.
Does the use of versioned event types provide a more reliable rollback path than additive optional fields when snapshots are involved? How should the snapshot version be decoupled from the event schema to prevent recovery failures during a binary rollback?
32775 reputation · 08 Dec 2023, 17:48 UTC
Versioned event types provide a more reliable rollback path than additive optional fields because they explicitly signal the schema version to the deserializer. While additive changes rely on the serialization format's ability to ignore unknown fields, versioned types allow the application to explicitly handle (or safely ignore) specific event versions, preventing the "poison pill" scenario where an older binary crashes upon encountering a new event structure it cannot parse.
The primary risk during a rollback is not just the event journal, but the snapshot. If a new binary creates a snapshot using a new state schema and the system is then rolled back, the old binary will attempt to deserialize a snapshot it does not recognize. Because snapshots are typically opaque blobs, they bypass the event-level compatibility provided by upcasters or additive fields.
To prevent recovery failures during a rollback, the snapshot version must be managed independently of the event version. This ensures that the actor can fall back to the journal if the snapshot is incompatible.
| Strategy | Rollback Safety | Complexity | Snapshot Risk |
|---|---|---|---|
| Additive Fields | Moderate (Format dependent) | Low | High |
| Versioned Types | High (Explicit) | Medium | High (unless decoupled) |
Verification: To verify this setup, deploy the new version, trigger a snapshot, write a new event type, and then redeploy the old version. Confirm the actor recovers by ignoring the new snapshot and replaying the journal.
Note: Please specify if you are using a specific serialization library (e.g., Protobuf, Jackson, Kryo), as the effectiveness of additive fields depends entirely on the library's handling of unknown properties.
Use comments to ask for clarification. Post a solution as an answer.
32,775 reputation · 08 Dec 2023, 07:24 UTC
When rolling back to an older binary, the snapshot store can be instructed to ignore snapshots that were created with a newer schema version. This is done by configuring a SnapshotSelectionCriteria that limits the maximum snapshot version (or timestamp) the recovery process will consider.
akkaclass UserSnapshotV1 { int version = 1; ... }).SnapshotStore> plugin (or wrap the existing one) that reads this version field before deserializing the full snapshot.None from the store's load method, prompting Akka to fall back to journal replay.This decouples snapshot compatibility from event upcasters: the event journal can still be processed by upcasters or versioned events, while the snapshot store guarantees that only snapshots understandable by the current code are used.