Correct Halo Exchange with Coarray Fortran: A Practical Example

A Fortran code that needs more than one core

You have a Fortran program that solves a 1‑D heat equation by sweeping over a one‑dimensional array. The mesh has grown beyond what a single core can handle in reasonable time, but you would rather avoid the boilerplate of MPI (matching sends/receives, linking an external library, rewriting subroutine interfaces). You want distributed‑memory parallelism while keeping the code looking like ordinary Fortran array assignments.

The useful takeaway: Coarray Fortran, standardized in Fortran 2008, lets you declare arrays with an extra trailing dimension in square brackets—real :: u(n, [*]). Each running copy of the program, called an image, gets its own storage for that dimension, yet any image can read or write another image's copy directly, as in u(:, 3). No explicit message‑passing calls are required.

How images see memory

An image is one instance of the compiled program. The intrinsic num_images() returns the total count, and this_image() returns the current image's index, from 1 to num_images(). Every image executes the same source code, but each works on its own local slice of a coarray unless it explicitly references another image.

A coarray declaration looks like this:

real, dimension(n, *) :: temperature

The asterisk inside brackets is the coarray dimension. Image 1 references its own storage with temperature(:, 1), and image 5 can read image 1's left boundary with temperature(1, 1). Because this is one‑sided communication, the remote image does not participate in the operation; it simply exposes its memory.

Synchronization is not optional

One‑sided access means writes to another image are not automatically visible, and there is no guaranteed ordering across images unless you insert a synchronization statement. sync all acts as a global barrier: no image proceeds past it until every image has arrived. sync images([2, 5]) synchronizes only the listed images, which is useful for neighbor exchanges in a domain decomposition.

Coarrays can be allocatable, derived‑type components, or multidimensional arrays, so a derived type can carry a coarray component for a distributed data structure. The trade‑off is that you, not the compiler, are responsible for ensuring that a remote write completes before another image reads that location.

Worked example: correct halo exchange on a decomposed domain

This program splits a 1D grid across images. Each image owns a interior slice plus two halo cells (one on the left, one on the right). After computing the interior update, each image writes its interior boundary values into the halo cells of its neighbours using one‑sided coarray references, then performs a global sync so that the halo data become visible. Finally each image updates its interior points using the halo values.

program heat_coarray_halo
  implicit none
  integer, parameter :: n_local = 400   ! interior points per image
  integer :: me, total, i
  ! coarray: interior + two halo cells (indices 0 and n_local+1)
  real, dimension(0:n_local+1, *) :: grid
  real :: left_halo, right_halo

  me = this_image()
  total = num_images()

  ! Initialise interior with an image‑dependent value; halo starts zero.
  grid(1:n_local, me) = real(me)
  grid(0, :) = 0.0
  grid(n_local+1, :) = 0.0

  ! Make sure every image has written its interior before neighbours read.
  sync all

  ! One‑sided write of interior boundary values into neighbour halos.
  if (me > 1) then
     ! send left boundary (index 1) to neighbour's right halo (index n_local+1)
     grid(n_local+1, me-1) = grid(1, me)
  end if
  if (me < total) then
     ! send right boundary (index n_local) to neighbour's left halo (index 0)
     grid(0, me+1) = grid(n_local, me)
  end if

  ! Ensure the halo writes are visible to all images.
  sync all

  ! Read halo values that now contain neighbour data.
  left_halo  = grid(0, me)
  right_halo = grid(n_local+1, me)

  ! Interior stencil update using halo values at the same index.
  do i = 2, n_local-1
     grid(i, me) = 0.25 * (grid(i-1, me) + grid(i+1, me) + left_halo + right_halo)
  end do

  sync all

  print *, 'Image', me, 'of', total, 'finished local update'
end program heat_coarray_halo

Compile with a coarray‑enabled compiler (e.g., gfortran, Flang, or Intel Fortran). The exact flag differs by compiler and version; a typical invocation for gfortran 12+ looks like:

gfortran -fcoarray=lib -o heat_coarray_halo heat_coarray_halo.f90
./heat_coarray_halo  --coarray-num-images=4

Expected check: each image prints a distinct index from 1 to the number of images you launched. If every line shows image 1, the launcher did not spawn multiple images or the compiler produced a single‑image executable. A risk of omitting the sync all after the halo writes is that an image may read stale or uninitialized halo data from its neighbours.

Trade‑offs and limitations

• Granularity matters. Frequent fine‑grained reads of remote images can be dominated by network latency. Coarser blocks of work usually scale better than element‑by‑element remote access.
• Debugging is distributed. A segmentation fault on one image may not abort the others immediately, which complicates post‑mortem analysis.
• Compiler runtime differs. Image‑launch mechanisms and environment variables are not portable between GCC, Flang, and Intel Fortran without checking documentation.
• Not a replacement for all MPI patterns. Irregular communication or dynamic task scheduling is often easier to express explicitly with messages.

Where to start

Before parallelising a real application, compile a minimal program that prints this_image() and num_images(), then launch it under your compiler's runtime image launcher with two or more images. Confirm that each image reports a unique index. Once that works, decompose your data into coarse blocks, add sync all where remote data must be visible, and measure whether coarse blocks or fine‑grained exchanges give better runtime for your problem.