Question
Evaluation limits of pandas.DataFrame.assign keyword arguments
Dev Moss
0 reputation · 29 Nov 2023, 01:48 UTC
55.2K views
Loading…
0 reputation · 29 Nov 2023, 01:48 UTC
Users often want to chain column creation in a single pandas.DataFrame.assign call, hoping that later keyword arguments can use columns added by earlier ones. The documented behavior, however, evaluates each keyword argument against the original DataFrame before any assignments take effect, so later keywords cannot see newly created columns and raise a KeyError.
This design contrasts with the intuitive left‑to‑right sequential expectation and has prompted frequent work‑arounds such as multiple assign calls or using pipe. Although the issue has been discussed in the pandas development community, the evaluation order has been retained through all releases up to at least pandas 2.2.0, indicating it remains an unresolved design decision rather than a bug.
Is the current evaluation order intended to stay permanent? Should pandas provide an explicit mode for sequential evaluation? What would be the impact on existing code if the order were changed?
35775 reputation · 29 Nov 2023, 09:48 UTC
In pandas.DataFrame.assign, all keyword arguments are evaluated against the original DataFrame before any new columns are created. This means that later keywords cannot reference columns added by earlier ones in the same call.
The evaluation order is intentional. It guarantees that assign is a pure transformation: the output depends only on the input DataFrame and the supplied values, not on the order of evaluation. This avoids hidden side‑effects and makes the method easier to reason about, especially when used in method‑chaining pipelines.
As of pandas 2.2.0 (and the discussion in the pandas development community), there is no announced plan to change the default evaluation order or to add a built‑in sequential mode. Introducing such a mode would require a new parameter (e.g., assign(..., eval='sequential')) or a separate method, and the core team has preferred to keep the API simple.
Switching to left‑to‑right sequential evaluation would be a breaking change for any existing code that relies on the current behavior, such as:
pandasdf.assign(new_col=df['old']*2, another=new_col+1) works now because new_col is not visible; changing the order would make another see the newly created new_col and produce a different result).If pandas ever provided an explicit sequential mode, existing code would continue to work unchanged; only users who opt‑in to the new mode would see the different behavior.
To confirm the current behavior in your environment, run:
import pandas as pd
df = pd.DataFrame({'a': [1, 2, 3]})
try:
df.assign(b=df['a']*2, c=df['b']+1)
except KeyError as e:
print('KeyError as expected:', e)
# Successful sequential version using two calls:
df2 = df.assign(b=df['a']*2).assign(c=lambda x: x['b']+1)
print(df2)
The first assign raises a KeyError because b is not yet available; the second approach shows the expected result when columns are added sequentially.
Use comments to ask for clarification. Post a solution as an answer.
6,690 reputation · 29 Nov 2023, 05:11 UTC
One clarification worth adding to the accepted answer: passing a callable doesn't change the evaluation context. In df.assign(a=1, b=lambda x: x['a'] + 1), the lambda receives the original DataFrame, so x['a'] still raises KeyError. Callables only defer evaluation—they don't make earlier kwargs visible.
A pattern that works in a single expression is nesting the dependency inside one callable chain:
df.assign(a=1).assign(b=lambda x: x['a'] + 1)If you want to avoid the extra assign call, you can compute the intermediate value outside and reuse it:
a = pd.Series(1, index=df.index)
df = df.assign(a=a, b=a + 1)Since both kwargs are evaluated against the original frame, referencing the precomputed a object is safe. Verified on pandas 2.2.x; worth re-checking the docs before relying on this in future releases, but no sequential mode has been announced.