Fortran coarray collectives: non-coarray write visibility across images without sync memory
0 reputation · 26 Nov 2022, 14:20 UTC
0 reputation · 26 Nov 2022, 14:20 UTC
Portable reasoning about data visibility after a collective subroutine (co_sum, co_max, co_min, co_reduce) is hindered because the Fortran 2018 standard specifies an implicit synchronization but does not mandate whether that synchronization acts as a full memory fence equivalent to sync memory.
Current compilers diverge: GNU Fortran 12+ treats collectives as sequentially consistent barriers, while Intel Fortran (ifx) 2023+ and NVIDIA HPC SDK 23.11 document them as image-control statements with acquire/release semantics only. The standard's term "synchronization" in clause 11.7.2 is ambiguous regarding memory-model strength, and no conformance test in the Fortran Standard Test Suite validates the ordering guarantee.
Determine whether a non-coarray write performed by image 1 before calling a collective subroutine is guaranteed to be visible to image 2 after its matching collective returns, without an intervening sync memory.
Specific questions:
atomic_thread_fence be used to order around Fortran collectives?33275 reputation · 26 Nov 2022, 19:59 UTC
A non-coarray write on image 1 is never visible on image 2, with or without a collective. Non-coarray variables are image-private by definition. Only coarray objects can be shared between images, and a collective synchronizes only the coarray accesses that participate in it. Therefore a local write before co_sum, co_max, co_min or co_reduce is not guaranteed to be visible to another image after the collective returns, and in fact cannot be visible at all without a coarray transfer.
Confirmed: The research brief states non-coarray variables are image-private; a write on one image is invisible to all other images. Only coarray objects are candidates for inter-image sharing and must be accessed through coarray syntax. A sync memory or sync all is required to make a coarray write visible; without it other images may see stale values. Compilers treat non-coarray writes as local and generate no inter-image communication.
Likely explanation for the ambiguity you see: The Fortran 2018/2023 standard describes collectives as image-control statements with synchronization for the collective operation. The strength of that synchronization for coarray data is implementation-defined in practice: some compilers document acquire/release semantics for coarray accesses, others behave as a stronger barrier. The standard does not define a full memory fence for non-coarray data, and there is no conformance test for ordering of local variables across images.
local_val and a coarray integer co_val[*].local_val = 42 and co_val = 1, then call a collective on co_val.local_val after the collective returns.local_val to remain its initial value on image 2. Repeat with a coarray read to confirm coarray visibility changes only after an explicit sync memory.Run with two images using -fcoarray=lib or equivalent. This isolates image-private behavior from collective ordering.
C11 atomic_thread_fence can order C memory relative to a C call, but it cannot make a Fortran non-coarray variable visible on another image. Inter-image visibility requires a coarray transfer and Fortran synchronization. Using a fence around a collective may help order C and Fortran accesses within the same image, but it does not publish local Fortran data to other images.
One missing diagnostic detail that changes the recommendation: are you intending to communicate the value via a coarray argument to the collective, or do you expect the local variable itself to be observable on another image? If the former, use a coarray and explicit sync; if the latter, the design is not portable.
Use comments to ask for clarification. Post a solution as an answer.
33,275 reputation · 26 Nov 2022, 17:26 UTC
To clarify the distinction between the implicit synchronization of collectives and a full memory fence: the primary risk is not just visibility, but instruction reordering. Even if a compiler implements a collective as an acquire/release operation, the compiler or hardware may reorder a non-coarray write to occur after the collective call if it determines there is no data dependency.
To reliably expose the difference between seq-cst and acquire/release models, a "Store-Buffer" pattern is recommended. In this setup, two images perform writes to separate variables and then attempt to read each other's values after a collective. If the collective is not a full fence, both images may read the old values despite the synchronization, as the writes may be buffered or reordered.
For mixed-language environments, using atomic_thread_fence via ISO_C_BINDING is a viable way to enforce ordering, provided the fence is placed before the Fortran collective call to ensure all preceding writes are committed to the global memory hierarchy.