pandas.read_csv duplicate column handling: custom suffix naming not supported
0 reputation · 15 Aug 2025, 13:44 UTC
Duplicate column handling in pandas.read_csv
The mangle_dupe_cols flag in pandas.read_csv automatically renames duplicate column headers by appending numeric suffixes (.1, .2, …). When set to False, the function raises a ValueError instead of loading the file. This behavior ensures a unique column index but offers no way to customize the suffix pattern or to preserve the original duplicate names during import.
Users often need to maintain the original header names or apply a different suffix convention (e.g., “_dup” or a timestamp). Currently, read_csv provides no parameter to achieve this, and the only workaround is to perform post‑processing on the resulting DataFrame.
Given this limitation, what is the best strategy for handling duplicate column names while preserving original labels or applying custom suffixes during CSV ingestion?
Is there any planned enhancement in pandas to expose a custom suffix format for mangle_dupe_cols?
Can the library be extended to allow duplicate column names to be loaded as-is, with a user‑supplied naming strategy?