Shared DuckDB database file compatibility across mixed engine versions

Question

Teams that share a single DuckDB database file across Python, R, and CLI environments face a storage-format compatibility policy decision. Because DuckDB is embedded, each environment bundles its own engine version. The v1.0.0 release is documented to introduce a stable storage format commitment for files created by that version and later, but opening a file written by a newer release with an older client is expected to raise an error rather than degrade gracefully. This leaves an unresolved design decision for mixed-version access.

The core uncertainty is whether to pin one DuckDB version across all readers and writers, or to standardize interchange on Parquet/CSV so the engine version and file format remain decoupled. Additional constraints include the possibility that a newer writer may upgrade the file in place, and the separate locking rules that apply when multiple processes access the same file.

Should a single DuckDB version be pinned across every environment touching the shared file, or is a decoupled interchange format more appropriate for mixed-version workflows? If a newer engine writes to the file in place, what migration path is supported when an older client must still read it? How can current storage-format compatibility boundaries be confirmed for the specific versions deployed in each environment?

Tasadduq Burney · Answer

Version Pinning vs. Decoupled Interchange
    For teams sharing a single .duckdb file across Python, R, and CLI environments, pinning a single DuckDB version across all environments is the only way to maintain direct file access. If your workflow requires flexibility in engine versions or supports clients that cannot be synchronized, a decoupled interchange format (such as Parquet) is the appropriate architectural choice.
    
The Compatibility Gap
    While DuckDB v1.0.0 introduced a stable storage format, compatibility is strictly backward, not forward. This creates a specific risk profile for mixed-version teams:
    
      Backward Compatibility: A newer engine (e.g., v1.1.0) can read a file created by an older engine (e.g., v1.0.0).
      Forward Compatibility: An older engine cannot read a file created or upgraded by a newer engine. This results in an \"Invalid database version\" error.
      In-Place Upgrades: Any process opening a database in read-write mode may upgrade the storage version on disk. Once upgraded, the file is permanently inaccessible to older clients.

Migration and Recovery Paths
    If a newer engine has already upgraded a shared file and an older client must read the data, there is no built-in \"downgrade\" command. The only supported migration path is a manual export/import cycle:
    
      Use the newer engine to export the required tables to Parquet or CSV: COPY table_name TO 'data.parquet' (FORMAT PARQUET);
      Use the older engine to import those files into a new database instance.

Verification and Enforcement
    To prevent accidental lockouts, verify the version alignment across your environments using these commands:
    -- Check the version of the engine currently running
PRAGMA version;

-- Check the storage version of the attached database file
PRAGMA storage_version;
    To enforce stability, use strict version pinning in your environment managers:
    
      Python: pip install duckdb==1.1.x
      R: remotes::install_version('duckdb', '1.1.x')
      CLI: Download the specific release binary corresponding to the pinned version.

Diagnostic Note: Please specify if any environments are accessing the file via a network mount (NFS/SMB), as this can introduce separate locking behaviors independent of version compatibility.

Shared DuckDB database file compatibility across mixed engine versions

1 answer

Version Pinning vs. Decoupled Interchange

The Compatibility Gap

Migration and Recovery Paths

Verification and Enforcement

Your perspective belongs here.

0 question comments