DuckDB skips unqueried columns in Parquet files

The problem: reading wide Parquet files costs more than it should

You have a Parquet file with hundreds of columns, but your query only needs three. You run the query anyway, memory usage climbs, and the result takes longer than it should. The useful takeaway is that DuckDB usually does not need to read every column: it prunes columns your query does not reference, and you can verify this with EXPLAIN before you change any infrastructure.

How column pruning works in DuckDB

DuckDB stores data column by column. When it reads Parquet or CSV, the query planner inspects the SELECT list, WHERE clause, JOIN keys, and aggregate expressions. Any column not referenced by those parts is marked as unused, and the Parquet reader skips its page and column chunks. This is different from SELECT *: if you ask for every column, nothing can be pruned.

Predicate pushdown works alongside pruning. Filters that DuckDB can evaluate directly on encoded column data—such as equality, range comparisons, and IN lists—are pushed below row materialization, so matching rows are identified before full rows are assembled. Row materialization is the step where DuckDB builds a complete row from the columns it has decoded.

Worked example: checking what DuckDB will read

Run this in the DuckDB CLI or a Python session with the duckdb package. You need read permission on the Parquet file; no write access is required because this is a read-only diagnostic.

EXPLAIN SELECT user_id, event_time
FROM 'events.parquet'
WHERE event_date = '2026-10-01';

Replace 'events.parquet' with your file path and the column names with your actual schema. The expected check is the plan output: look for a scan operator that lists only the referenced columns plus any columns needed for the filter. If event_date appears in the filter but not the SELECT list, it should still appear in the scanned columns because the planner needs it to evaluate the predicate.

Risk: EXPLAIN does not execute the query, so it will not reveal runtime errors such as missing files or type mismatches. Use EXPLAIN ANALYZE if you want actual timings, but that will read data.

When pruning does not help

• Queries with SELECT * reference every column, so nothing is skipped.
• User-defined table functions or custom scan functions can bypass DuckDB's columnar reader entirely.
• Non-SARGable expressions—such as wrapping a column in a function inside WHERE, for example WHERE UPPER(status) = 'ACTIVE'—may prevent predicate pushdown and force more column reads.
• Nested subqueries or complex joins can obscure column usage from the planner in some edge cases.

Practical verification

1. Run EXPLAIN on your query and note the scanned columns in the plan.
2. Compare the plan against a version of the query that uses SELECT *; the starred version should list more columns.
3. If the plan still shows all columns, simplify the query: avoid SELECT *, inline functions on filtered columns, and unnecessary subqueries.
4. Check memory usage in the DuckDB client while running the query on a wide file versus a narrow projection to confirm the runtime benefit.

Closing

Column pruning is transparent—you do not rewrite queries to enable it—but it is not automatic magic. The fastest way to confirm it is working is to inspect the plan with EXPLAIN. If your workload scans wide Parquet files and touches only a few columns, pruning can reduce I/O and memory meaningfully; if you use SELECT * or opaque functions in filters, you will likely read the whole table anyway.