Cut pandas Memory and Speed Up groupby with Categorical Dtypes

You have a column like status or country_code that repeats the same few strings across millions of rows. Pandas stores each value as a separate Python string object, and every groupby has to compare and move that text. Converting the column to Categorical changes the storage model: pandas keeps the unique strings once and replaces each row with a compact integer code. For low-cardinality data, this usually lowers memory use and makes aggregations faster.

Why repeated strings are expensive

When you load text into a default pandas column, the dtype is object. Under the hood, pandas stores an array of pointers to Python str instances. Even if the same word appears a million times, the column carries a million references, and operations like groupby must hash and compare the full string contents unless pandas can optimize the path. The larger the row count, the more this overhead matters.

How Categorical changes the model

A Categorical column splits data into two parts: a small list of unique categories and an array of integer codes that point into that list. Instead of repeating "active" a million times, pandas stores "active" once and uses a small integer for every row. This reduces the bytes that must be scanned during a groupby and improves cache locality because the working set is smaller.

You can inspect the internals directly:

df["status"].dtype          # category
df["status"].cat.categories # Index(['active', 'closed', ...])
df["status"].cat.codes      # integer codes, one per row

A worked example

Run the following in a Python interpreter or Jupyter notebook. No special permissions are needed; replace status with your target column if you are adapting it.

import pandas as pd
import numpy as np
import time

# Create a low-cardinality string column
n = 1_000_000
rng = np.random.default_rng(42)
values = rng.choice(["active", "pending", "closed", "archived"], size=n)
df = pd.DataFrame({"status": values})

# Measure baseline memory
before = df.memory_usage(deep=True).sum()

# Convert to categorical
df["status"] = df["status"].astype("category")

# Measure memory after conversion
after = df.memory_usage(deep=True).sum()

print(f"Before: {before:,} bytes")
print(f"After:  {after:,} bytes")

Expected checks: the after total should be noticeably lower than before when the row count is large and the number of unique strings is small. Confirm the dtype changed with df["status"].dtype.

Next, compare aggregation speed. Build two identical frames, convert one, and time the same groupby:

df_obj = pd.DataFrame({"status": values})
df_cat = df_obj.copy()
df_cat["status"] = df_cat["status"].astype("category")

t0 = time.perf_counter()
df_obj.groupby("status").size()
t1 = time.perf_counter()

t2 = time.perf_counter()
df_cat.groupby("status").size()
t3 = time.perf_counter()

print(f"object groupby:   {t1 - t0:.4f}s")
print(f"category groupby: {t3 - t2:.4f}s")

On a low-cardinality column, the categorical groupby is typically faster because pandas operates on integer codes rather than string objects. On very small frames the difference may be within noise.

Trade-offs and limitations

Categorical is not a universal win. Keep these limits in mind:

• High cardinality hurts. If the number of unique values is close to the number of rows, the categories index plus the codes array can consume more memory than the original strings. A quick diagnostic is df[col].nunique() / len(df); if that ratio is high, skip the conversion.
• Sorting semantics change. By default, sort_values() on a categorical column uses the order of the categories, not lexical (alphabetical) order. If your workflow depends on string sorting, either sort before converting or explicitly define an ordered category list.
• Conversion has a cost. Building the category index takes time and temporary memory. For a one-time aggregation on a small dataset, the overhead may outweigh the benefit.

Aspect	object dtype	category dtype
Storage	Full string per row	Unique categories + integer codes
Best for	High-cardinality text	Low-cardinality repeated text
Default sort	Lexical	Category order
Memory with repeats	Higher	Lower

Actionable takeaway

Before converting, check the cardinality ratio. If a string column has few distinct values relative to its length—say, under a few percent—and it participates in memory-heavy or aggregation-heavy workflows, run df[col] = df[col].astype("category"). Verify the win with df.memory_usage(deep=True).sum() and a timed groupby. If you need the original strings back, df[col] = df[col].astype(str) restores the object dtype. This behavior is consistent across modern pandas 1.x and 2.x releases.