Process Large CSV Files with pandas chunked reading
Learn how to iterate over a massive CSV file with pandas' read_csv(chunksize=) to keep memory usage low while computing aggregates.
Loading…
ReadMeFeed / Community knowledge
#csv-processing
Real questions. Useful conversations. Find the people who know your stack.
pandasFast CSV Lookups with Awk Associative Arrays
Learn how to use awk's associative arrays to perform fast, memory-efficient CSV joins and lookups directly from the command line without needing a database.
awkSplitting Delimited Strings in Rexx Using the PARSE Statement
Learn how to use the Rexx PARSE statement to efficiently split delimited strings into named variables, including handling empty fields and dynamic field counts.
rexxStreaming Large CSVs with pandas: A Chunked Aggregation Architecture
A practical architecture for processing large CSV files with pandas without loading the entire dataset into memory, including schema gates, incremental aggregation, and operational checks.
Processing CSVs Larger Than RAM: A pandas Chunking and Memory-Mapping Guide
Process multi-gigabyte CSV files in pandas without exhausting RAM by combining chunked iteration with memory-mapped I/O, explicit dtypes, and incremental aggregation.