Query Parquet Files Directly in DuckDB Without an ETL Step

Skip the ETL step for one-off Parquet analysis

You receive a directory of Parquet exports from an application log or a data pipeline. The business question is simple: how many events failed on October 18, and which users triggered them? Loading that data into a warehouse just to run one query takes longer than the analysis itself.

DuckDB can query Parquet files directly. You point SQL at a file path, and the engine reads only the columns and row groups the query needs. It is not a replacement for a transactional database, but for read-only ad-hoc analytics it often removes the need for an ETL load step entirely.

Inspect the schema before you scan

Run this in the DuckDB CLI or from a Python client:

DESCRIBE SELECT * FROM read_parquet('events/2024-10-18.parquet');

Replace events/2024-10-18.parquet with your actual path. You need read permission on the file or directory. DESCRIBE returns column names and types without reading the full file, which is useful when Parquet files are produced by systems with evolving schemas.

To verify that DuckDB is pushing filters into the Parquet reader rather than scanning everything, use EXPLAIN:

EXPLAIN SELECT user_id, status
FROM read_parquet('events/2024-10-18.parquet')
WHERE status = 'failed' AND event_ts >= '2024-10-18 00:00:00';

Look for PARQUET_SCAN in the plan with filter expressions listed beneath it. If the plan shows a full scan followed by a separate filter operator, pushdown may not have applied. Check your DuckDB release notes for the installed version, because pushdown behavior and supported expression types change between releases.

Query partitioned directories and many files at once

If your files are laid out in Hive-style partitions—year=2024/month=10/day=18/—DuckDB can discover the directory structure and expose year, month, and day as virtual columns:

SELECT day, count(*) AS failures
FROM read_parquet('events/year=2024/month=10/*.parquet', hive_partitioning = true)
WHERE status = 'failed'
GROUP BY day
ORDER BY day;

You can also pass a list of paths or a glob pattern to scan sharded files in one query. This behaves like a union across files without creating intermediate tables.

Globbing many small files adds file-open overhead. If you control the file layout, fewer larger files with bigger row groups usually scan faster than thousands of tiny files.

Materialize results only when repetition justifies it

Direct Parquet reads are read-only. You cannot run UPDATE or row-level DELETE against a Parquet file through DuckDB. If you need to enrich, join repeatedly, or modify data, load it into a DuckDB table first:

CREATE TABLE events_daily AS
SELECT * FROM read_parquet('events/2024-10-18.parquet');

To undo this, run DROP TABLE events_daily;.

Or share a filtered result set:

COPY (SELECT * FROM read_parquet('events/2024-10-18.parquet') WHERE status = 'failed')
TO 'failed_events.parquet' (FORMAT PARQUET);

To undo a COPY, delete the output file. Materializing creates a new object, so keep the original Parquet files intact and treat the copy as a derived artifact.

Trade-offs and how to check them

• Read-only analytics: Direct Parquet scans do not support updates or deletes. Practical check: attempt an UPDATE on a read_parquet query and expect an error; if you need mutations, load into a DuckDB table.
• Memory on wide or nested files: Even when you select a few columns, deeply nested types or very wide schemas can increase memory during projection. Check with DESCRIBE before running broad queries.
• Small-file overhead: Time an identical filtered query against a directory of many small Parquet files and against one compacted file. Do not assume the glob query is faster.
• Dialect differences: DuckDB SQL differs from PostgreSQL and Spark SQL. Some functions or syntax may not exist. Verify functions against the documentation for your installed version.

Start with DESCRIBE, then decide whether to load

For ad-hoc questions over Parquet files, begin with DESCRIBE SELECT * FROM read_parquet('your-file.parquet') to confirm the schema, then run your query with EXPLAIN to confirm filter pushdown. If you find yourself running the same filter daily, or if you need to update rows, copy the data into a DuckDB table. Otherwise, skip the ETL step and query the files where they sit.