Pandas read_csv low_memory flag causes unexpected dtype promotion and memory spikes

Direct answer

When low_memory=False is set, pandas reads the whole file to infer column dtypes before storing the data. If a column contains mostly integers but also a few NaN values, the inference sees that the integer dtype cannot represent missing values and promotes the column to float64 (the default float type that can hold NaN). The resulting DataFrame therefore uses more memory per cell (8 bytes vs. 4 or 1‑2 bytes for smaller integer types) and the inference step itself may allocate temporary arrays, causing a noticeable memory spike.

Confirmed facts

• The low_memory flag only influences the dtype‑inference pass; it does not change the final data unless a promotion occurs.
• With low_memory=True (default) pandas reads the file in chunks, infers dtypes from the first few rows, and may later promote columns object‑wise if later rows disagree.
• Setting low_memory=False disables the chunk‑based guess and forces a single‑pass inference over the entire file, but the data is still loaded into memory after inference.
• Pandas 1.3 and 2.x behave the same way regarding this flag; no documented change in the memory‑allocation strategy for low_memory=False between those versions.

Likely explanation for the observed spike

1. During the full‑file inference pass pandas builds a temporary NumPy array for each column to examine all values.
2. When it encounters a NaN in an integer‑looking column, it decides the column must be float, allocates a new float64 array, and copies the data.
3. The original integer array may remain alive briefly until the reference is swapped, temporarily doubling the memory needed for that column.
4. After inference, the final DataFrame holds the float64 column, which consumes more memory than the original integer representation would have.

Steps to verify and mitigate

1. Inspect a small sample to see the raw values:

   import pandas as pd
   df_sample = pd.read_csv('your_file.csv', nrows=1000)
   print(df_sample.dtypes)
   print(df_sample['problem_column'].head())
   

2. Read the full file with low_memory=False and measure memory:

   df = pd.read_csv('your_file.csv', low_memory=False)
   mem = df.memory_usage(deep=True).sum()
   print(f'Memory usage: {mem / 1e6:.2f} MB')
   print(df.dtypes['problem_column'])
   

3. If the column should stay integer, either fill the missing values with an integer sentinel (e.g., -1) or explicitly specify the dtype:

   df = pd.read_csv('your_file.csv',
                    dtype={'problem_column': 'Int64'},   # nullable integer
                    low_memory=False)
   

Missing diagnostic detail

To confirm that the promotion is solely due to NaN values and not other non‑integer entries, could you share a few sample values from the problematic column (e.g., the first 10 non‑null entries)? Knowing whether any strings or decimals appear would determine whether specifying a nullable integer dtype (Int64) is sufficient or whether the column must stay as object.