Diagnosing Memory Ordering Issues on AArch64

Code that is stable on x86_64 can show intermittent stale reads on AArch64 because the architecture allows the hardware and compiler to reorder memory operations for performance. The useful takeaway: a visible flag does not guarantee the data it protects is visible without explicit ordering.

Recognizable condition: the stale data race

The classic symptom is a producer that writes a buffer and then sets a ready flag, while a consumer sees the flag true but reads old or partially written data. The failure is non-deterministic and typically appears only under load on ARM64 hardware.

Ordered checks

1. Confirm architecture-specific behavior

Run the same build on x86_64 and AArch64 with identical inputs and a multi-threaded stress harness. A behavioral difference points to memory ordering rather than logic. No elevated permissions are required, only access to both targets.

2. Locate flag-data patterns

Search for shared variables written by one thread and read by another without a mutex. Pay attention to a store to a flag that follows stores to data. Flag any site using plain stores or relaxed atomics.

3. Inspect generated assembly

On the build host, with read access to the binary, use objdump to verify compiler ordering:

objdump -d your_binary | grep -A 30 <function_name>

Replace your_binary and <function_name> with your target. Look for a DMB or an acquire/release load-store between the data store and the flag store. Absence suggests the compiler reordered or elided a barrier. Risk: objdump output is large; filter to the specific function to avoid misreading unrelated code.

4. Stress test with ThreadSanitizer

Compile with ThreadSanitizer on an AArch64 target and run the stress harness:

clang -fsanitize=thread -g -O1 -o test test.c

TSan reports data races that are logically possible even if not observed in a single run. Use enough iterations to increase coverage. Limitation: TSan adds overhead and changes timing, so a clean run is evidence, not proof.

Fixes tied to findings

Missing release-acquire on the flag

If the check shows a plain store for the flag, replace it with acquire-release atomics. In C++ this is std::atomic with memory_order_release on the producer store and memory_order_acquire on the consumer load. The release store pairs with the acquire load so prior writes become visible to the reader.

Incorrect lock-free exclusive access

When building lock-free structures, use Load-Exclusive/Store-Exclusive with a retry loop. The store succeeds only if no intervening access occurred:

loop:
    LDXR   W0, [X1]      @ load exclusive
    ADD    W0, W0, #1
    STXR   W2, W0, [X1]  @ store exclusive, status in W2
    CBNZ   W2, loop      @ retry on failure

Do not insert non-exclusive accesses between LDXR and STXR. Verify the retry loop exists in the compiled output, since compilers can restructure intrinsics.

Barrier choice

AArch64 provides DMB, DSB, and ISB. DMB orders memory accesses relative to the barrier without halting execution. DSB is stronger and waits for completion, which can degrade performance if overused. ISB flushes the instruction pipeline and is needed after self-modifying code, not for data ordering. Use the weakest barrier that satisfies the ordering requirement.

Limitation: DMB does not guarantee immediate global visibility, only ordering relative to the barrier on the issuing core. Do not treat it as a cache flush to all observers.

Verification

After a change, re-run the AArch64 stress harness and compare behavior to x86_64. Inspect the assembly again to confirm the barrier or acquire-release codegen is present. Monitor performance to detect regressions from added DSB instructions.

Escalation criteria

If races persist after correct acquire-release usage and proper LDXR/STXR loops, suspect a logic error in the state machine or access to non-coherent memory. Replace the lock-free path with a mutex-based critical section as a diagnostic test. If the mutex eliminates the symptom, the issue is ordering; if it persists, investigate data races elsewhere or memory attributes. Escalate to architecture review when non-coherent regions or device memory are involved.