When pandas Categoricals Actually Pay Off: Memory Savings and Faster groupby

If a string column in your DataFrame repeats the same few values millions of times, converting it to pandas' category dtype will usually cut memory use substantially and can make groupby aggregations faster. The catch: the benefit depends entirely on cardinality. Convert a low-cardinality column and you win; convert a near-unique column and you add overhead for nothing.

How the category dtype works

Internally, a categorical column stores each distinct value once in a categories array, then stores the data as small integer codes pointing into that array. A column of 10 million rows with 20 distinct strings becomes 20 strings plus 10 million small integers (int8 or int16 depending on category count), instead of 10 million Python string references. That is where both the memory savings and the groupby speedup come from: groupby can operate on the integer codes rather than hashing strings.

A worked example

Run this in any Python environment with pandas installed (pandas 1.x or 2.x; behavior described here is stable across both). No special permissions are needed — this is in-memory and changes nothing on disk.

import pandas as pd

# Suppose df has a 'region' column with a handful of repeated strings
print(df['region'].nunique())          # check cardinality first
before = df.memory_usage(deep=True).sum()

df['region'] = df['region'].astype('category')

after = df.memory_usage(deep=True).sum()
print(f"Memory: {before/1e6:.1f} MB -> {after/1e6:.1f} MB")

result = df.groupby('region', observed=True)['revenue'].mean()

The deep=True argument to memory_usage matters: without it, pandas reports only the pointer size for object columns, hiding the real cost of the strings. Note observed=True in the groupby — without it, pandas includes unobserved category combinations in the result (and in older versions emits a FutureWarning about the default changing).

When it helps and when it hurts

The rule of thumb is cardinality relative to row count. If the number of unique values is well under half the row count, categoricals typically win. The savings grow as cardinality shrinks. If nearly every row has a distinct value, the codes array adds a second data structure without deduplicating anything, so memory can actually increase and groupby gains nothing.

Scenario	Categorical worth it?
Status column, 5 values, 1M rows	Yes — large memory cut, faster groupby
Country column, ~200 values, 10M rows	Usually yes
User ID, 900K unique in 1M rows	No — overhead without dedup benefit
Free-text comments	No

Common mistakes

• NaN handling. Missing values are not a category. In .cat.codes they appear as -1, which silently corrupts any arithmetic you do on codes. If you need NaN as a group, add it explicitly with df['region'] = df['region'].cat.add_categories(['unknown']).fillna('unknown').
• String methods. The .str accessor works on categoricals but returns an object-dtype result, so chained string operations silently undo your conversion. Do string cleanup first, convert to category last.
• Merges. Merging a categorical column against a non-categorical column of the same name can upcast the result back to object, erasing the savings. Convert both sides, or re-convert after the merge and verify with df.dtypes.
• Stale categories after filtering. Filtering rows does not shrink the category set; groupby may still emit empty groups for categories that no longer appear. Use df['region'] = df['region'].cat.remove_unused_categories() after heavy filtering.
• .apply surprises. Some apply paths cast the column back to object. Check df.dtypes after transformations rather than assuming the dtype survived.

Limitations

Adding new, previously unseen values to a categorical forces pandas to rebuild the codes array, which is expensive on large frames — for high-churn data, convert once after the data settles, not incrementally. Also, while groupby on codes is generally faster for low-cardinality data, the speedup is workload-dependent; some operations see no measurable difference, and a few paths are slower because of category bookkeeping.

How to verify the benefit

Don't convert on faith. Measure both dimensions:

# Memory: run in your notebook or script
print(df.memory_usage(deep=True).sum())   # before and after astype('category')

# Speed: time the actual aggregation you care about
%timeit df.groupby('region', observed=True)['revenue'].mean()

A clear drop in total memory confirms the encoding took effect. For speed, compare the same aggregation on the object-dtype and categorical versions of the column; keep the change only if the categorical version is equal or faster on your real data. If neither metric improves meaningfully, revert with df['region'] = df['region'].astype(str) — the conversion is cheap to undo and leaves no persistent state.