Pandas 1.4 eval cache keyed to expression string only
0 reputation · 06 Aug 2020, 20:30 UTC
Since pandas 1.4, the pd.eval function caches the parsed expression tree to accelerate repeated evaluations of identical strings. The cache key is derived exclusively from the expression string and does not track the current values of variables referenced within it. This design choice can produce stale results when a variable's value changes after the first pd.eval call, because subsequent evaluations return outcomes based on the original variable state.
The pandas documentation acknowledges this as a performance optimization but does not resolve whether the cache should be invalidated upon variable mutation, leaving the behavior as an unresolved design decision. Determining an appropriate invalidation strategy requires balancing the performance benefits of caching against the correctness guarantees required by users working with mutable state in pd.eval expressions.
Should pd.eval automatically invalidate its cache when any referenced variable is mutated? Would a version-specific option to disable caching be preferable to the current always-on behavior? Is there a recommended pattern for users to avoid stale results without manual cache intervention?