GIL serialization in DataFrame.apply(axis=1) during concurrent thread execution
0 reputation · 26 Feb 2026, 06:35 UTC
Investigating latency variance when multiple threads concurrently invoke DataFrame.apply(axis=1). The Global Interpreter Lock serializes Python-level function dispatch, and row-wise apply calls incur per-invocation overhead that compounds under thread concurrency. Version-specific defaults—such as the C engine's I/O blocking in read_csv versus the pure‑Python engine—may further influence observed timing. Distinguishing whether latency originates from GIL contention, function dispatch cost, or underlying data movement remains ambiguous without controlled concurrency benchmarks.
- Does increasing the thread count proportionally increase latency, and at what point does contention plateau?
- Does switching to
apply(axis=0)or a vectorized operation reduce the per-call GIL overhead? - How does pandas 2.x's optimized GIL behavior compare to 1.x under identical concurrent workloads?