DataFrame.append removal in pandas 2.0 and concat replacement
0 reputation · 29 Feb 2020, 13:39 UTC
0 reputation · 29 Feb 2020, 13:39 UTC
Migration of legacy code that used DataFrame.append and Series.append from pandas 1.x to pandas 2.0+ crosses a documented compatibility boundary. The append methods were deprecated in pandas 1.4 and removed in pandas 2.0, so row-wise combination now relies on pandas.concat.
The change shifts from a method style API to a function that returns a new object and aligns columns by name, filling missing values with NaN. The migration assumes control over the pandas runtime and needs to preserve row order, index continuity, and dtype stability across concatenated frames with potentially different schemas.
Uncertainty remains about semantics when the original code relied on append's default behavior for indexes and column union.
What semantics does pandas.concat provide for ignore_index compared to the former append default? How does column alignment and NaN filling differ from append when schemas diverge? Are there dtype upcasting risks when concatenating frames with mixed dtypes that append previously tolerated?
In pandas 2.0+, DataFrame.append and Series.append are removed; calls raise AttributeError. The supported row-wise replacement is pd.concat(frames, axis=0).
ignore_index semantics are effectively unchanged from the old append default. Both default to False, which preserves original index labels and permits duplicates. Setting ignore_index=True in either API replaces the index with RangeIndex(0, n) and discards the old labels.
For column alignment, concat(axis=0) performs an outer union of columns by name, as append did. Rows missing a column are filled with NaN for object or numeric frames, or pd.NA for nullable extension dtypes. There is no behavioral difference in the basic alignment rule.
Dtype upcasting is the main migration risk. Concatenating integer columns with missing values typically widens the column to float unless nullable dtypes such as Int64 are used. Mixing int and float yields float; object, categorical, timezone-aware datetime, and mixed extension columns may unify to a broader dtype or object. append tolerated similar coercion, but concat results must be explicitly checked and cast.
| Item | Status | Detail |
|---|---|---|
| append removed in pandas 2.0 | Confirmed | Deprecated in 1.4; removed in 2.0. |
| ignore_index default | Confirmed | False in both append and concat. |
| outer column union | Confirmed | Missing values filled with NaN or NA. |
| verify_integrity replacement | Likely explanation | concat has no verify_integrity; check index.duplicated() manually. |
df.append(other) with pd.concat([df, other], axis=0, ignore_index=False).ignore_index=True only if the legacy code reset the index.pd.concat once; do not concat repeatedly inside a loop.index.duplicated().any(), column set, isnull().sum(), and dtypes against the target schema.verify_integrity=True, raise or drop duplicates when index.duplicated().any() is true.import pandas as pd
print(pd.__version__) # expect 2.0+
frames = [df_left, df_right]
result = pd.concat(frames, axis=0, ignore_index=False)
print(result.index.duplicated().any())
print(result.dtypes)
print(result.isnull().sum())
One missing diagnostic detail changes the recommendation: did the legacy code pass verify_integrity=True to append? If yes, add the duplicate-index check shown above; otherwise the basic concat replacement is sufficient.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 29 Feb 2020, 15:33 UTC
One subtle migration detail: pd.concat defaults to sort=True in pandas 1.x but sort=False in pandas 2.0+. This means column order preservation differs from append unless you explicitly set sort=False when upgrading. The verify_integrity parameter (default False) also behaves differently—it raises on duplicate indexes in concat whereas append silently allowed them. For strict schema validation during migration, consider pd.concat([df1, df2], ignore_index=True, sort=False, verify_integrity=True) to catch index collisions early.