All-NaN slices in np.nanmean: limits of the RuntimeWarning policy and distinguishing 'no data' from a NaN result
0 reputation · 09 Aug 2020, 13:03 UTC
0 reputation · 09 Aug 2020, 13:03 UTC
I am designing a data pipeline on NumPy (assume a recent 1.2x/2.x release) that reduces float arrays with np.nanmean and np.nansum along an axis. Some slices may be entirely NaN, and I need downstream code to treat "no valid observations" differently from "valid data whose mean happens to be NaN".
The documented behavior is that NaN-aware reductions skip NaNs, but an all-NaN slice emits a RuntimeWarning and returns NaN. As far as I can tell there is no parameter on these functions to change that policy — the only lever seems to be the global or scoped warning machinery (np.seterr / np.errstate or the warnings module), which treats the warning as noise to suppress rather than as a signal I can capture per slice.
My constraints: this runs inside library code, so mutating global error state with np.seterr is not acceptable, and I would prefer to avoid a separate pre-pass with np.isnan over every slice if the API already offers a cleaner hook.
Is there any documented way to configure or intercept the all-NaN-slice behavior of np.nanmean directly, rather than filtering warnings? If not, is counting valid elements separately (e.g., via np.sum(~np.isnan(x), axis=...)) the intended pattern, and does it carry a meaningful performance cost compared with the reduction itself? Finally, does np.errstate reliably scope warning handling in threaded code so two libraries do not interfere?
A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.