Question
JetStream snapshot restore compatibility with NATS 2.10 backup format headers
Zo Byte
0 reputation · 22 Feb 2024, 16:20 UTC
65K views
Loading…
0 reputation · 22 Feb 2024, 16:20 UTC
When restoring a JetStream stream from a snapshot backup, the interaction between the backup format version and the target server release creates a compatibility boundary affecting stream data and consumer state. Snapshots created with NATS 2.10 include a version header in the backup format, and the nats-restore tool is documented to restore snapshots only to the same or a newer NATS version. This creates a practical uncertainty for operators maintaining mixed-version clusters or staging restore targets.
Beyond the format header, snapshot backups capture stream data and metadata but omit consumer state unless explicitly exported. There is currently no automated mechanism to re-create consumer state during restore, so consumers must be recreated manually after the stream is available. Additionally, restoring to a server with a different cluster ID will orphan the stream, requiring the cluster ID to match before restore.
Given these constraints, what is the supported path for restoring a NATS 2.10 snapshot to a target server running an earlier release, and does the version header prevent the restore outright or merely signal a format change? Is there a documented workflow for exporting and re-importing consumer configurations alongside a stream snapshot when the restore target predates automated consumer state handling? How should operators verify that restored stream data and any manually recreated consumers match the original backup without relying on undocumented restore behavior?
35275 reputation · 22 Feb 2024, 19:10 UTC
There is no supported path for restoring a NATS 2.10 snapshot to a target server running an earlier release. The version header in the backup format is a hard guard; it prevents the restore process from proceeding if the target server version is lower than the version that created the snapshot. This is designed to prevent data corruption or metadata mismatches that occur when a server lacks the logic required to parse the 2.10 format.
While it may seem like a simple metadata check, the version header signals potential changes in how the underlying stream data is structured or how the nats-restore tool interacts with the server API. Attempting to bypass this header via manual file manipulation is unsupported and risks silent data loss or server instability.
Because snapshots capture stream data and metadata but omit consumer state (such as ack floors and offsets), you must handle consumer recovery as a separate operational step. Since there is no automated mechanism to bundle consumer state within the snapshot, use the following workflow:
To ensure the restored environment matches the original backup without relying on undocumented behavior, perform these three checks:
natsnats stream info [STREAM_NAME] on both source and target to verify that message counts and sequence numbers are identical.nats stream ls to ensure stream configuration (retention, discard policies) is consistent.nats consumer info [STREAM] [CONSUMER] to validate that manually recreated consumers are correctly bound to the stream and are ready to deliver messages.Required Diagnostic: To provide a more specific recovery path for your consumers, please specify if you are using Push or Pull consumers, as the manual recreation of sequence positions differs between the two.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 23 Feb 2024, 00:54 UTC
To build on the restore workflow, it is critical to highlight the role of the Cluster ID. While the version header prevents restores to older NATS releases, a matching Cluster ID is required for the restored stream to be correctly integrated into the JetStream metadata of the target cluster.
If a snapshot from NATS 2.10 is restored to a target server with a different Cluster ID, the stream may become orphaned. This means that while the data exists on disk, the JetStream management layer may not recognize it as part of the current cluster's active set, leading to failures when attempting to create consumers or publish new messages.
Verification Step: Before initiating the restore, verify the Cluster ID on both source and target using the NATS CLI or server logs. If they differ, the target cluster configuration must be updated to match the source before the nats-restore process begins to ensure the stream is properly attached.