DuckDB to Apache Parquet round-trip type fidelity for TIMESTAMPTZ and INTERVAL
0 reputation · 02 Feb 2022, 15:27 UTC
0 reputation · 02 Feb 2022, 15:27 UTC
Determine whether DuckDB preserves TIMESTAMP WITH TIME ZONE and INTERVAL columns through a write-to-Parquet then read-from-Parquet cycle without changing the SQL type or losing values.
DuckDB offers read_parquet() and COPY ... TO ... (FORMAT PARQUET), but INTERVAL and TIMESTAMP WITH TIME ZONE lack an exact Parquet primitive. They may be written as annotated or converted logical types, and read_parquet() infers schemas by sampling files. Exact annotations vary by DuckDB release, and external consumers can strip metadata.
For the installed DuckDB release, what behavior should be expected at this integration boundary?
35775 reputation · 02 Feb 2022, 21:08 UTC
The installed DuckDB release does not automatically restore the original TIMESTAMP WITH TIME ZONE or INTERVAL types when a Parquet file is read with read_parquet() without an explicit schema; TIMESTAMP WITH TIME ZONE is inferred as plain TIMESTAMP (zone offset lost) and INTERVAL is inferred as a string or numeric type, discarding interval semantics.
Note: behavior may differ between DuckDB versions prior to 1.2, where INTERVAL serialization is limited. Please provide the exact DuckDB version installed to confirm whether interval handling is fully supported.
Use comments to ask for clarification. Post a solution as an answer.
6,230 reputation · 02 Feb 2022, 16:03 UTC
When writing a table with TIMESTAMP WITH TIME ZONE or INTERVAL, use COPY ... TO 'file.parquet' (FORMAT PARQUET) SCHEMA to embed the exact types at write time; this prevents loss if downstream tools strip metadata. After writing, read with read_parquet('file.parquet') and, if needed, supply an explicit schema via the schema argument to guarantee correct type mapping. Verify fidelity by comparing typeof(column) and sample values before and after the round‑trip.