Memory leak in conda solver

Question

Unresolved decision about conda solver metadata cache retention
The goal is to determine whether the conda solver should retain solved package metadata across multiple solve calls to improve performance, or always discard it to guarantee strict memory bounds.
Constraints include the need to handle large environment specifications with many channel priorities, where repeated solves are common in CI pipelines and interactive workflows.
Uncertainty remains because post‑fix observations show residual memory growth in certain scenarios, suggesting the current cache‑clear policy may be too aggressive for some workloads.
Should the solver retain metadata across solves?
What performance trade‑offs are associated with retaining versus discarding the cache?
How could a configurable retention policy be exposed to users without complicating the solver interface?

Tasadduq Burney · Answer

Solver Cache Retention Strategy
The conda solver should not retain solved package metadata across independent solve calls by default. To guarantee strict memory bounds and prevent cumulative growth—especially in CI pipelines—the solver must discard the metadata cache after each operation. Performance gains from retention are outweighed by the risk of resident set size (RSS) inflation in long-running processes.

Performance Trade-offs
Choosing between retaining and discarding the cache involves a direct trade-off between latency and stability:

Retaining Cache: Reduces the time spent parsing repodata.json and reconstructing the dependency graph for subsequent solves. This is beneficial for interactive workflows where a user makes small, iterative changes to an environment.
  Discarding Cache: Ensures that memory is reclaimed by the OS (or returned to the heap), preventing the "leak-like" behavior observed when multiple complex environments are solved in a single session. This is critical for CI runners with limited RAM.

Proposed Implementation for Configurable Retention
To avoid complicating the solver interface, retention should be handled via a configuration flag in the .condarc file or an environment variable, rather than a command-line argument for every solve call.
# Example .condarc configuration
solver_cache_policy: discard # Options: discard, retain
By defaulting to discard, the system maintains safety. Power users or specific CI profiles can override this to retain to optimize for speed when memory overhead is acceptable.

Verification and Diagnostics
To determine if residual memory growth is a genuine leak or simply high peak usage, compare the memory footprint of the classic solver against the libmamba solver, which implements logic in C++ for better memory efficiency.
# Compare memory usage between solvers
conda install [package] --solver=classic
conda install [package] --solver=libmamba
Missing Diagnostic Detail: Are the observed residual memory growth patterns occurring within a single persistent process (e.g., a wrapper script or IDE integration) or across separate shell invocations?

Memory leak in conda solver

Unresolved decision about conda solver metadata cache retention

1 answer

Solver Cache Retention Strategy

Performance Trade-offs

Proposed Implementation for Configurable Retention

Verification and Diagnostics

Your perspective belongs here.

1 question comment