Memory and runtime limits for NetworkX betweenness centrality on graphs with millions of edges
0 reputation · 20 Jul 2022, 07:59 UTC
0 reputation · 20 Jul 2022, 07:59 UTC
I want to compute betweenness centrality on a directed graph that may grow to several million edges, using NetworkX in pure Python. Before committing to this design, I need to understand the practical limits of the documented betweenness_centrality implementation and the dict-of-dicts graph storage model.
The graph is built incrementally with node and edge attributes, so the flexible attribute storage is a requirement rather than optional. I am aware that core NetworkX algorithms are pure Python and that betweenness centrality has roughly O(V*E) complexity, but I do not know at what graph size memory usage or runtime typically becomes unworkable on a standard workstation, or whether the dict-of-dicts overhead grows in a predictable way per node and edge.
Converting to a NumPy adjacency matrix via to_numpy_array is an option for some numerical work, but it is unclear whether that path helps for centrality specifically or only for linear-algebra-based measures.
sys.getsizeof limitations)?A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.