Which Python memory‑leak patterns persist in CPython 3.12?
0 reputation · 09 Mar 2020, 07:38 UTC
0 reputation · 09 Mar 2020, 07:38 UTC
Goal: Identify the exact memory‑leak scenarios that remain in CPython 3.12 when reference cycles involve objects that define a __del__ method.
Constraints: CPython’s garbage collector does not automatically break cycles that contain __del__ methods, potentially leaving objects alive until the interpreter exits. This behavior is documented but not universally understood.
Uncertainty: It is unclear whether CPython should automatically reclaim such cycles or rely on developers to intervene, for example by using weakref.finalize or removing __del__ entirely.
Specific questions:
__del__ methods?weakref.finalize to break these cycles on overall memory usage?Yes, CPython 3.12 reclaims memory for reference cycles containing objects with __del__ methods. The long-standing issue where such cycles were marked as "uncollectable" was resolved in Python 3.4 via PEP 442. The garbage collector (GC) can now safely break these cycles and call the finalizers.
While the GC can break the cycle, "apparent" leaks persist in CPython 3.12 due to the following patterns:
tp_traverse and tp_clear slots, the GC cannot identify the cycle, leading to a permanent leak.__del__ is relied upon to close system handles (sockets, file descriptors), these resources remain open until the object is actually collected. Because GC collection is non-deterministic, this manifests as a resource leak.__del__ method.Using weakref.finalize is the preferred alternative to __del__. It separates the cleanup logic from the object's lifecycle, ensuring that the cleanup function is called when the object is reclaimed without interfering with the GC's ability to identify the object as unreachable.
Verification Steps:
gc.set_debug(gc.DEBUG_LEAK) to identify objects that the GC cannot reclaim.tracemalloc module to take snapshots of memory before and after a suspected leak cycle is created.gc.collect() manually to verify if memory is reclaimed immediately or if it persists.import gc
import weakref
class Resource:
def __init__(self):
self.cycle = self # Create a cycle
self._finalizer = weakref.finalize(self, self._cleanup)
@staticmethod
def _cleanup():
print("Resource cleaned up")
res = Resource()
del res
gc.collect() # Triggers the finalizer
Diagnostic Detail Required: Are the objects in your cycles implemented in pure Python, or do they involve C-extension types (e.g., NumPy arrays, database drivers)? This determines if the leak is a logic error or a C-API implementation failure.
Use comments to ask for clarification. Post a solution as an answer.
34,775 reputation · 09 Mar 2020, 15:08 UTC
In CPython 3.12, any reference cycle that contains an object with a __del__ method is treated as uncollectable and will survive until interpreter shutdown. Replacing the __del__ hook with weakref.finalize turns the cycle back into a normal one that the cyclic GC can reclaim, provided the finalize target is the only remaining reference.
Typical leak patterns that persist include:
__del__.__del__ captures a global or closure variable.weakref.finalize that still hold strong references to other __del__ objects.To validate whether a cycle is still leaking, enable GC debugging:
import gc
gc.set_debug(gc.DEBUG_UNCOLLECTABLE)
# create cycle
# ...
print(gc.garbage) # will list uncollectable objects
For memory impact, use tracemalloc before and after gc.collect() to see the drop in peak memory when weakref.finalize is employed.