All-NaN slices in np.nanmean: limits of the RuntimeWarning policy and distinguishing 'no data' from a NaN result
0 reputation · 09 Aug 2020, 13:03 UTC
0 reputation · 09 Aug 2020, 13:03 UTC
I am designing a data pipeline on NumPy (assume a recent 1.2x/2.x release) that reduces float arrays with np.nanmean and np.nansum along an axis. Some slices may be entirely NaN, and I need downstream code to treat "no valid observations" differently from "valid data whose mean happens to be NaN".
The documented behavior is that NaN-aware reductions skip NaNs, but an all-NaN slice emits a RuntimeWarning and returns NaN. As far as I can tell there is no parameter on these functions to change that policy — the only lever seems to be the global or scoped warning machinery (np.seterr / np.errstate or the warnings module), which treats the warning as noise to suppress rather than as a signal I can capture per slice.
My constraints: this runs inside library code, so mutating global error state with np.seterr is not acceptable, and I would prefer to avoid a separate pre-pass with np.isnan over every slice if the API already offers a cleaner hook.
Is there any documented way to configure or intercept the all-NaN-slice behavior of np.nanmean directly, rather than filtering warnings? If not, is counting valid elements separately (e.g., via np.sum(~np.isnan(x), axis=...)) the intended pattern, and does it carry a meaningful performance cost compared with the reduction itself? Finally, does np.errstate reliably scope warning handling in threaded code so two libraries do not interfere?
No: as of recent NumPy 1.2x/2.x releases, np.nanmean and np.nansum expose no parameter to change the all-NaN-slice policy. An all-NaN slice returns NaN and emits a RuntimeWarning ("Mean of empty slice" / "All-NaN slice encountered"), and the only API-level lever is the warning/error machinery. The documented, intended pattern for your use case is a separate valid-count pass, and it is the only reliable discriminator anyway — because a NaN result does not imply an all-NaN slice (e.g., inf - inf inside the sum also yields NaN from valid data).
count = np.sum(~np.isnan(x), axis=axis)
result = np.nanmean(x, axis=axis)
no_data = count == 0 # True exactly where the slice had no observations
result[no_data] = sentinel # or keep a separate mask downstreamThis is explicit, vectorized, library-safe (no global state touched), and gives you a per-slice "no data" channel — something warning capture fundamentally cannot do, since a warning tells you that an all-NaN slice occurred, not which one.
The count pass is roughly one extra traversal of the array (isnan + sum), so total work is about 2× the bare reduction. Both passes are memory-bandwidth-bound C loops, so for most pipelines the overhead is real but rarely dominant. If profiling on your array sizes shows it matters, consider np.ma.mean on a masked array: all-masked slices come back as masked rather than NaN, giving you the "no data" channel in a single pass at the cost of masked-array machinery and its own overhead. Measure both on representative data rather than assuming.
np.errstate is a context manager that saves and restores the ufunc error state, and that state is thread-local in NumPy — so scoping it inside a thread is safe and two libraries in different threads will not interfere. It is the correct tool for library code, unlike process-global np.seterr. But note what it does: it only converts the warning to ignore/warn/raise. np.errstate(invalid='raise') turns the condition into a FloatingPointError that aborts the entire reduction — useful as a fail-fast assertion, not for per-slice diagnosis. And warnings.catch_warnings(record=True) mutates process-global filter state and is not safe for concurrent capture across threads, so it is a poor fit for your constraint.
Exact warning messages and whether nanmean routes through the invalid error bucket have varied across NumPy 1.x/2.x. Verify on your installed version:
import warnings, numpy as np
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter('always')
np.nanmean(np.full((2, 3), np.nan), axis=1)
print([str(x.message) for x in w])And confirm the count mask matches np.isnan(result) on mixed data before relying on it.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.