Should OCaml's Marshal module include a version header for forward‑compatible backup restoration?

Short answer

Adding a version header directly to the Marshal format would give a clear forward‑compatibility marker, but it also introduces the risk of breaking every existing marshaled blob. In practice, the safest path is to keep the current format unchanged and add a lightweight wrapper that carries the version (and, optionally, word‑size or checksum) outside the Marshal payload. This preserves backward compatibility while still providing the desired safety guarantees.

Why a direct header change is risky

• The Marshal binary layout is defined per compiler release and contains no version field. Inserting a byte before the payload would shift all offsets that older deserialisers rely on, causing Marshal.Error or, worse, silent corruption.
• Existing blobs produced by older OCaml runtimes would become unreadable by the new runtime unless every consumer is also upgraded.
• Because the Marshal format is not portable across 32‑/64‑bit architectures, a header cannot solve word‑size mismatches unless it explicitly encodes that information.

What a minimal version indicator could look like

Conceptually, a single‑byte version number inserted immediately after the current four‑byte magic number would add at most four bytes of overhead. The payload would remain unchanged, so existing serialisers would still produce the same data after the magic bytes. A reader that understands the new header could simply skip the byte if it sees the magic, while older readers would treat the byte as part of the payload and therefore fail.

Impact on size and speed

• Size: Adding 1–4 bytes to a typical backup (kilobytes to megabytes) is below 0.5 % overhead, generally negligible for storage or network transfer.
• Speed: The extra read and a couple of integer comparisons are insignificant compared to the cost of serialising or deserialising the payload itself.

Migration strategy without breaking existing blobs

1. Maintain two readers: one that expects the old format (no header) and one that can parse the new header. The new reader should be able to fall back to the old reader when it encounters a blob without a header.
2. Deploy the new reader first. It accepts both legacy and new blobs, so existing consumers continue to work.
3. When all consumers are upgraded, switch writers to emit the new header (e.g., version 1). The new blobs can now include a word‑size field or checksum if desired.
4. If a future format version is required, the reader can reject blobs with an unknown version number, providing a clear error instead of silent data loss.

Alternative: an outer wrapper

Rather than changing Marshal itself, you can wrap the Marshal payload in a small envelope containing a magic string, a version byte, and optional metadata. The envelope is independent of the Marshal layout, so it can coexist with any existing blob. Readers that know the envelope can strip it before calling Marshal.from_bytes; readers that do not recognize the envelope simply treat the data as a raw Marshal blob. This approach preserves total backward compatibility and keeps the change isolated to your application code.

What we still need to decide

• Which OCaml compiler versions will you use for reading and writing? If you are confined to a single major version, the risk of incompatibility is lower, making a direct header more feasible.
• Do you need to support cross‑architecture (32‑/64‑bit) restores? If so, the header or wrapper should include a word‑size field or you must enforce that restores happen on the same architecture.
• Will the data ever be consumed by non‑OCaml tools? If yes, a Marshal‑only approach may be unsuitable regardless of headers.

Conclusion

In short, a version header is technically possible and would give a clear forward‑compatibility signal, but changing the Marshal format carries a high risk of breaking legacy blobs. A lightweight wrapper that adds a magic string, a version byte, and optional metadata is a safer, backward‑compatible alternative that still provides the intended safety guarantees. The choice depends on your specific versioning, architecture, and consumer constraints.