Query Parquet Files Directly with DuckDB Without Importing Them

The problem: importing a whole lake to answer one question

You have a Parquet export sitting on disk—maybe 20 GB of click events, maybe a monthly extract from a warehouse. Your stakeholder asks for a count by region for one day. The familiar path is to create a table, load the file, run the query, then drop the table. That path reads every column and every row before your filter ever executes. For ad-hoc local analytics, the import step is often the slowest part.

DuckDB can query the file where it lives

DuckDB exposes read_parquet as a table function. It opens Parquet files directly from the file system without copying them into a DuckDB table first. Because DuckDB runs in-process—inside the CLI, a Python script, or an embedded application—there is no separate database server and no network roundtrip. You point the function at a file or glob pattern and run SQL immediately.

Projection and pushdown mean less disk I/O

Parquet stores data column by column. When your query asks for only two columns, DuckDB performs column projection: it reads just those column chunks from disk. When your query includes a WHERE clause, DuckDB attempts predicate pushdown, sending the filter into the scan so Parquet metadata can skip row groups that cannot match. The result is that large files often cost far less I/O than a full table scan would suggest.

Verify the plan before you trust it

Pushdown behavior depends on the DuckDB release and the Parquet file's metadata, so do not assume it happened. Run your query through EXPLAIN and look for a PARQUET_SCAN operator that lists the projected columns and any pushed-down filters.

Where to run: DuckDB CLI or a Python session with the duckdb package.

Required permissions: read access to the Parquet file and its parent directory.

Placeholder: replace /path/to/events.parquet with your actual file path.

Expected check: the EXPLAIN output should reference only the columns in your SELECT list, not every column in the file.

Risk: selecting many wide columns or running a large aggregation can still consume substantial memory. DuckDB is optimized for analytical read workloads, not long-running multi-user transactional systems.

Worked example: events partitioned by date

Suppose your files are arranged in Hive-style folders:

data/events/
  year=2024/month=10/day=01/part-0.parquet
  year=2024/month=10/day=01/part-1.parquet
  year=2024/month=10/day=02/part-0.parquet

You can query the whole month as one logical table:

SELECT
  region,
  count(*) AS purchases
FROM read_parquet('data/events/year=2024/month=10/day=*/*.parquet')
WHERE event_type = 'purchase'
GROUP BY region
ORDER BY purchases DESC;

DuckDB discovers the partition folders and exposes year, month, and day as columns you can filter or group by. Because the filter on event_type can be pushed into the scan, DuckDB may avoid reading row groups that contain no purchases.

To confirm column projection, compare two queries on the same file. One selects a single narrow column; the other selects every column. Use EXPLAIN on both and confirm the single-column plan projects fewer fields. Do not copy the plan from another system or version; inspect it on your target DuckDB release.

Limitations and a practical check

• Glob expansion and Hive partition discovery are version-sensitive. Verify on the exact DuckDB release you plan to deploy.
• DuckDB is single-writer for persistent tables. Multiple processes writing to the same Parquet file concurrently is unsafe.
• Memory grows with wide scans and large aggregations. Start with a small file or a limited date range before scanning a full lake.

A practical way to check the result is to run EXPLAIN on your query, then run SELECT count(*) against the same read_parquet path and confirm the row count matches your expectations from the file metadata. If you need a curated artifact, use COPY (SELECT ...) TO 'output.parquet' (FORMAT PARQUET) to materialize a smaller file for downstream work.

Closing

Start with one file. Query it with read_parquet, inspect the plan with EXPLAIN, and confirm that only your selected columns and matching row groups are read. If the plan looks right, you have a lightweight local analytics loop without an import step or a database server.