Free‑GIL build: compatibility of legacy C‑extensions
0 reputation · 27 Nov 2024, 05:08 UTC
0 reputation · 27 Nov 2024, 05:08 UTC
Python 3.13 offers an experimental "free‑GIL" build that disables the Global Interpreter Lock, allowing true parallel execution of bytecode across cores. The primary goal is to evaluate whether legacy C‑extensions—many of which were written assuming the GIL provides atomicity—remain reliable when that guarantee is removed.
Constraints include the absence of a clear deprecation schedule for the standard GIL build, the increased reference‑counting overhead in the free‑GIL interpreter, and the potential for race conditions in extensions that perform non‑atomic operations on shared state. The behavior of these extensions under heavy concurrency is currently unverified.
What C‑extensions are documented to fail or exhibit race conditions in the free‑GIL build? What mitigation strategies (e.g., re‑entrancy wrappers, thread‑local storage) can be applied without rewriting the extension? When might the free‑GIL build receive a stable release or a deprecation plan for the standard GIL?
32775 reputation · 27 Nov 2024, 10:19 UTC
CPython does not publish a catalog of named legacy C extensions that fail in the 3.13 free‑threaded build. The documentation describes compatibility classes by code pattern, not by package. The likely explanation for failures is that many extensions implicitly rely on the GIL to serialize access to shared C state—global caches, borrowed references, refcounts, or non‑reentrant libraries—so removing the GIL exposes latent races.
Confirmed: the free‑threaded interpreter uses a separate ABI; extensions must be rebuilt against free‑threaded headers. The module definition can declare Py_mod_gil = Py_MOD_GIL_NOT_USED, but that is an opt‑in assertion of thread safety, not a compatibility flag. The build also changes reference‑counting behavior, adding overhead that is separate from correctness risks.
Likely explanation: extensions with static mutable state, lazy initialization, direct refcount manipulation, or calls into non‑thread‑safe C libraries are the highest risk. Extensions with no shared state and single‑threaded entry points are lower risk, but still unverified until stress‑tested.
| Pattern | Risk in free‑GIL |
|---|---|
| Global/static mutable state | High — races on shared C data |
| Lazy initialization caches | High — double init or torn writes |
Borrowed PyObject* after GIL release | High — lifetime violations |
| Non‑reentrant C library calls | High — internal library state corruption |
| No shared state, single‑threaded entry | Lower — still needs rebuild and stress test |
If you can rebuild, treat the free‑threaded ABI as a separate target and audit globals, caches, and borrowed PyObject* lifetimes before opting in to Py_MOD_GIL_NOT_USED. Note that Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS are not a free‑threading fix; object access and lifetime rules still apply.
There is no confirmed stable release date for free‑threading and no announced deprecation schedule for the standard GIL build. Python 3.13 should be treated as experimental for production C‑extension compatibility decisions. Check the current PEP 703 status and release notes for your exact target version before committing.
One diagnostic detail changes the recommendation: does the extension spawn internal threads or call back into Python? If yes, prefer process isolation or a real thread‑safety port over a wrapper lock.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.