Pandas 1.4 eval cache keyed to expression string only

In pandas 1.4+, pd.eval() caches the compiled expression tree using the expression string as the key. To avoid stale results when referenced variables are mutated, you must ensure the execution phase receives the updated environment or bypass the cache entirely.

Understanding the Cache Mechanism

It is important to distinguish between the compiled expression tree and the evaluated result. The cache stores the parsed logic (the "how"), not the final output (the "what"). Under normal operation, pd.eval() should use the current values of the variables provided in the local or global namespace during the execution phase.

Potential Causes of Stale Results

If you are observing stale results despite variable mutation, it is likely due to one of the following:

• Namespace Mismatch: The variables being mutated are not the same objects being passed to the local_dict or global_dict during the evaluation.
• Engine-Specific Behavior: The numexpr engine (the default for many numeric operations) may handle memory alignment or caching differently than the python engine.

Recommended Patterns to Avoid Stale Results

Since there is currently no global toggle to disable the expression tree cache in pandas 1.4, use these strategies to ensure correctness:

1. Explicitly Define the Namespace

Avoid relying on implicit global lookups. Pass a dictionary to the local_dict parameter to ensure the evaluator is looking at the most recent version of your data.

import pandas as pd

df = pd.DataFrame({'a': [1, 2]})
# First call
res1 = pd.eval("df.a + 1", local_dict={'df': df})

# Mutate data
df['a'] = [10, 20]

# Second call with explicit namespace
res2 = pd.eval("df.a + 1", local_dict={'df': df})

2. Use the 'python' Engine for Complex State

If numexpr is producing unexpected results during mutation, switch to the python engine. This bypasses the specialized numeric optimizations of numexpr and uses standard Python evaluation logic.

pd.eval("df.a + 1", engine='python')

3. Dynamic Expression Strings

If you must completely bypass the cache, appending a unique identifier (like a timestamp or counter) to a comment within the string will force a re-parse, as the cache key is the string itself.

# Forces a cache miss by changing the string key
pd.eval(f"df.a + 1 # {timestamp}")

Verification Step

To determine if your specific environment is suffering from a cache-related bug or a namespace issue, run the following test:

1. Execute pd.eval("df.a + 1").
2. Modify df in place.
3. Execute pd.eval("df.a + 1", engine='python').

If the python engine returns the correct value while the default engine does not, the issue is isolated to the numexpr backend's interaction with the cached tree.

Diagnostic Detail Needed: Are you using the default numexpr engine or have you explicitly set engine='python'?