Efficiently Query Large Parquet Files in Python Using DuckDB
Query large Parquet files directly in Python using DuckDB's SQL interface, avoiding memory errors associated with loading entire datasets into Pandas.
Loading…
ReadMeFeed / Community knowledge
#parquet
Real questions. Useful conversations. Find the people who know your stack.
duckdbQuery Parquet Files Directly in DuckDB Without an ETL Step
DuckDB can read Parquet files directly with SQL, pushing filters into the file reader for ad-hoc analytics without an ETL load step.
Query Parquet Files Directly with DuckDB Without Importing Them
Query Parquet files on disk with DuckDB's read_parquet, using column projection and predicate pushdown to read only what your SQL needs.
DuckDB Storage Strategy: Native .db Files vs. Parquet
A decision guide for choosing between DuckDB's native .db storage and Apache Parquet, focusing on ACID compliance, I/O performance, and data portability.
Exporting Partitioned Parquet Datasets from DuckDB with COPY TO
Learn how to write query results to a Hive-style partitioned Parquet dataset using DuckDB's COPY TO, validate the output, and recover from common write failures.
DuckDB read_parquet: When to Query Parquet in Place and When to Materialize
DuckDB's read_parquet lets you query Parquet lakes without ETL, but per-query metadata overhead and schema drift have real costs. Here's how to decide.
How DuckDB’s Vectorized Engine and Parquet Push‑Down Speed Up Analytical Queries
DuckDB’s vectorized engine and Parquet push‑down cut CPU work and I/O for analytical scans, as shown in a step‑by‑step example that creates a 100‑million‑row Parquet file, runs a filtered aggregation, and verifies push‑down with EXPLAIN.
DuckDB's Native Parquet Support: What Engineers Actually Need to Know
DuckDB's built-in Parquet extension lets you read and write columnar files directly from SQL with automatic predicate push-down and partitioning—no Spark cluster required. Here's the practical map for engineers moving data in pipelines.
Using DuckDB’s Native Parquet Engine for Zero‑Config Analytics
Learn how DuckDB lets you read, transform, and write Parquet files directly from SQL without extra services or complex setup.