Fortran Coarrays: Architecture Note on Requirements, Minimal Design, and Failure Modes

The Problem: Distributed Memory Without Explicit Message Passing

Fortran coarrays give you a single-program-multiple-data (SPMD) model where each image (process) owns its memory and can read or write variables on other images using a codimension syntax—real :: x[*]; x[2] = 5.0. The promise is simpler code than MPI for many stencil, halo-exchange, or reduction patterns. The reality is that the abstraction leaks at the runtime boundary: you still need a compliant compiler, a coarray runtime (usually OpenCoarrays over MPI or SHMEM), and a launch environment that starts the correct number of images. This note captures the architectural decisions you make before writing the first sync all.

Requirements That Shape the Design

• Compiler baseline: Fortran 2008 coarray support. GCC ≥ 9 (with -fcoarray=lib), Intel Fortran ≥ 2018, or NAG Fortran. Older GCC 4.9–8 have partial support but miss allocatable coarrays and sync images.
• Runtime library: OpenCoarrays (default MPI backend) or a vendor SHMEM layer. The runtime handles image startup, remote memory registration, and collective synchronization.
• Launch model: ./a.out 4 or CAF_NUM_IMAGES=4 ./a.out for OpenCoarrays; mpirun -np 4 ./a.out for Intel -coarray=distributed. The image count is fixed at launch—no dynamic spawn in the standard.
• Memory model: Each image has a symmetric heap. Remote accesses are one-sided (RDMA-style) but ordered only by explicit sync calls.

Smallest Viable Design

Start with two coarrays:

• A scalar integer :: ready[*] for point-to-point or collective synchronization.
• One or two allocatable coarrays for the actual problem data, e.g., real, allocatable :: field[:,:][*].

Allocate once after sync all so every image has the same shape. Keep the symmetric heap small—large allocations on every image can exhaust the per-process virtual memory limit imposed by the MPI runtime.

program minimal_coarray
  implicit none
  integer :: ready[*]
  real, allocatable :: field[:,:][*]
  integer :: me, nimg

  me = this_image()
  nimg = num_images()

  allocate(field(64,64))
  field = me * 1.0
  ready = 1
  sync all

  if (me == 1) then
    print *, 'Image 1 sees field(1,1) on image 2:', field(1,1)[2]
  end if
  sync all
  deallocate(field)
end program minimal_coarray

Compile and run (OpenCoarrays on Linux):

# Build
$ gfortran -fcoarray=lib -O2 minimal_coarray.f90 -o minimal_coarray
# Run with 2 images
$ CAF_NUM_IMAGES=2 ./minimal_coarray
# Expected output:
# Image 1 sees field(1,1) on image 2: 2.000000

Where to run: Any node with the compiler and OpenCoarrays installed. Permissions: User-level; no root needed. Placeholders: Replace 64,64 with your problem dimensions. Check: Program prints the remote value without ERROR STOP or hang. Risk: If CAF_NUM_IMAGES doesn't match the actual launch count, the runtime aborts with an image-count mismatch error.

Trust and Data Boundaries

Each image trusts only its own local memory. Remote accesses are explicit: x[target_image] appears in source, so a code review can enumerate every cross-image dependency. No implicit global address space exists. This property makes coarrays auditable for safety-critical codes—unlike PGAS models that expose a unified pointer space.

Boundary checklist:

• All coarray allocations use the same shape on every image (enforced by sync all before allocate).
• Cosubscripts are validated at runtime; out-of-bounds triggers ERROR STOP with image ID.
• No remote pointers escape the lexical scope of the coarray variable.

Operational Checks You Can Automate

1. Image count verification: At startup, assert num_images() == expected_count and ERROR STOP otherwise.
2. Cosubscript bounds: Compile with -fcheck=bounds (GCC) or -check bounds (Intel). The runtime inserts checks on every [idx] access.
3. Synchronization health: Wrap sync all in a timer; if it exceeds a threshold, log a warning—stalled images often indicate a missed sync images or a failed collective.
4. Failed-image detection: Fortran 2018 adds failed_images() and sync images(*, stat=ierr). Check ierr == STAT_FAILED_IMAGE and re-distribute work.

Example guard:

integer :: ierr
sync all
if (num_images() /= 4) error stop 'Launch with 4 images'
call cpu_time(t0)
sync images(*, stat=ierr)
call cpu_time(t1)
if (ierr /= 0) print *, 'Sync failed:', ierr
if (t1 - t0 > 5.0) print *, 'Sync latency high:', t1-t0

Failure Modes and How They Manifest

Mode	Symptom	Root Cause	Mitigation
Image count mismatch	Immediate abort from the coarray runtime	CAF_NUM_IMAGES ≠ launcher -np	Enforce single launch script; read num_images() at runtime.
Remote out-of-bounds	ERROR STOP with an image ID	Index calculation error, or shape differs across images	Compile with bounds checking; allocate after sync all.
Deadlock	Process hangs in sync all or sync images	One image never reaches the sync (crash, infinite loop, or mismatched sync images sets)	Timeout watchdog; sync images(*, stat=) with STAT_FAILED_IMAGE.
Memory overrun	SIGSEGV inside MPI/RDMA layer	Writing past coarray bounds corrupts symmetric heap metadata	Bounds checking; valgrind on single-image run.
Performance collapse	Runtime far slower than an MPI equivalent	Fine-grained remote accesses in tight loops; MPI eager/rendezvous threshold	Pack data into contiguous coarrays; use sync all once per timestep.

When the Design Must Change

Coarrays are a host-side coordination layer. Shift the architecture when:

• GPU offload is required. Current standard has no device coarrays. Use DO CONCURRENT with !$omp target on each image, keeping coarrays only for host-to-host halo exchange.
• Task parallelism outweighs data parallelism. If the workload is irregular (e.g., adaptive mesh refinement), MPI+OpenMP or a task runtime (OpenMP tasks) gives finer control than the SPMD image model.
• Latency-sensitive collectives. For sub-microsecond synchronization, vendor SHMEM (Cray, Intel) or UCX-backed runtimes beat the default MPI-based OpenCoarrays.
• Dynamic process groups. Fortran 2018 team_type and form team help, but if you need frequent group creation/destruction, MPI communicators are more mature.

Limitations and Verification Checklist

• No standard way to query remote image memory usage—rely on OS tools per rank.
• Debuggers (gdb, TotalView) typically attach to one image; remote state requires printf logging or MPI-aware debugger setups.
• Coarray performance is opaque: the same source can run over MPI, SHMEM, or GASNet with wildly different latency profiles.

Practical verification after each compiler/runtime upgrade:

1. Run the minimal program above with CAF_NUM_IMAGES=2 and 4.
2. Run a 1D halo-exchange benchmark (1000 iterations, 1 MB per image) and measure sync all time.
3. Inject a deliberate out-of-bounds access on image 2; confirm the runtime aborts with a clear image ID.
4. Kill one image with SIGKILL mid-run (if the runtime supports failed-image detection) and verify the surviving images receive STAT_FAILED_IMAGE.

If any step fails, the coarray stack—not your algorithm—is the blocker.