Pandas DataFrame.merge duplicate column handling ambiguous
18.7K reputation · 16 Feb 2021, 19:37 UTC
Pandas DataFrame.merge duplicate column handling
When merging two DataFrames that share column names outside the join keys, pandas automatically appends suffixes via the suffixes parameter. However, the exact handling—whether the first occurrence is preserved, a warning is issued, or an error is raised—has not been explicitly documented for all pandas releases. The indicator flag adds a _merge column, but the string values for row origin may differ between minor releases. This uncertainty affects downstream data pipelines that rely on predictable column names and merge metadata.
Questions
- What is the default behavior of pandas DataFrame.merge when both inputs contain duplicate column names that are not join keys?
- Does pandas raise an exception, keep the first column, or rename automatically, and has this changed in recent releases?
- How do the
_mergeindicator values vary across pandas versions, and what guidance exists for handling them?