Cutting Memory Use in Pandas with Categorical Data Types

Memory pressure and the categorical solution

When a DataFrame stores many repeated strings, each row keeps a full copy of the text, which quickly inflates RAM usage. This hampers loading speed and can limit the size of data you can process. In practice, a column of city names stored as plain objects can consume several megabytes even for a modest dataset. When the same value appears many times, storing it repeatedly wastes space, and operations like groupby or sort_values can become slower because they must handle duplicate strings.

The categorical dtype solves this by replacing each string with an integer code and storing the actual strings once in a lookup table. For columns where the same values recur, memory consumption can drop by up to 90 percent.

Convert, verify, and measure

Steps

1. Ensure pandas is imported in your Python environment.
2. Apply the conversion: df['column_name'] = df['column_name'].astype('category'). Replace 'column_name' with the actual column label.
3. Confirm the dtype change with df['column_name'].dtype; it should read category.
4. Check memory usage before and after using df.memory_usage(deep=True). You will typically see the categorical column use 80‑90 percent less memory than the original object column.

Running these commands requires only the ability to execute Python code; no elevated permissions are needed. The example below demonstrates the transformation on a small dataset.

After conversion, you can also inspect the categories with df['column_name'].cat.categories to verify that the mapping reflects the expected unique values. Because categorical data are stored as integers, operations such as groupby or sort_values can be faster, especially on large datasets.

import pandas as pd

data = {'city': ['Paris', 'Berlin', 'Paris', 'Tokyo', 'Berlin', 'Paris']}
df = pd.DataFrame(data)

print("Before conversion")
print(df['city'].dtype)  # object
print(df.memory_usage(deep=True).sum())

df['city'] = df['city'].astype('category')

print("\nAfter conversion")
print(df['city'].dtype)  # category
print(df.memory_usage(deep=True).sum())

Expected outcome: the dtype line shows category and memory usage drops, typically by 80‑90 percent for this pattern.

When categories aren't enough

Categorical dtype shines when the number of distinct values is modest and stable. If a column has high cardinality—many unique strings that change frequently—the memory savings diminish and the overhead of maintaining the mapping can increase. In such cases, keeping the column as plain object may be more efficient, and you might consider other compression techniques instead. A common rule of thumb is to avoid categorical conversion when the number of unique values approaches the total number of rows.

Actionable closing

Identify string columns that repeat often, convert them to categorical dtype, re‑measure memory, and decide if the performance gain outweighs the added complexity. For columns with high cardinality or frequent updates, stay with the default object dtype. Verifying the memory impact with memory_usage(deep=True) before and after the conversion is the safest way to proceed.