Diagnosing Memory Ordering Bugs when Porting to ARM64

The Problem: The 'Works on x86' Fallacy

Code that has run stably for years on x86-64 often fails intermittently when ported to ARM64 (aarch64). These failures typically manifest as rare data corruption, null pointer dereferences in supposedly initialized objects, or crashes that only appear in release builds on high-core-count machines like AWS Graviton or Apple Silicon.

The cause is the difference in Memory Consistency Models. x86-64 uses Total Store Order (TSO), a strong model where the hardware guarantees that stores are not reordered with other stores. ARM64 uses a Weakly Ordered model. On ARM64, the CPU and the compiler are free to reorder memory operations to improve performance, provided the result is consistent for a single thread. In multithreaded environments, this means one core may see a "ready" flag set to true before it sees the data that the flag is supposed to protect.

Symptom and Cause Mapping

Symptom	Likely Cause	Technical Detail
Intermittent crash on first access to a shared object	Publication Race	Store to object data and store to "ready" pointer reordered.
Stale data read despite a lock being released	Missing Acquire Barrier	Load of data occurred before the load of the lock state.
Data corruption in lock-free queues/stacks	Insufficient Fencing	Lack of explicit memory barriers (DMB) between dependent stores.
Consistent failure in MMIO/Driver code	Device Ordering Issue	CPU reordered writes to hardware registers.

Diagnostic Workflow

Follow these steps in order. Do not skip to assembly review until higher-level race detection is exhausted.

1. Race Detection (TSan): Compile your application with ThreadSanitizer (-fsanitize=thread). TSan detects data races by monitoring memory accesses. If TSan reports a race on a variable used for synchronization, you have a correctness bug regardless of the architecture.
2. Stress Testing: Run the application on physical ARM64 hardware with a thread count equal to or exceeding the physical core count. Use a loop that iterates millions of times. Virtual machines often mask ordering bugs due to different timing characteristics.
3. Audit Publication Patterns: Search for "flag-based" synchronization. Any pattern resembling if (ready) { use(data); } where ready is a plain boolean or integer is a bug on ARM64.
4. Assembly Verification: For critical paths, use objdump -d or Compiler Explorer. Look for the instructions used for atomic operations. On ARM64, LDAR (Load-Acquire) and STLR (Store-Release) are the standard for acquire/release semantics. If you see plain LDR/STR for shared synchronization variables, the code is unsafe.

Fixing Ordering Violations

The fix depends on the finding. Never use volatile to fix these issues; volatile prevents compiler optimization but does not generate the hardware barriers needed for ARM64.

Scenario: The Publication Race

Incorrect (x86-style):

// Thread A
shared_data = 42;
ready = true; // Plain store

// Thread B
if (ready) {
    print(shared_data); // May print 0 or crash on ARM64
}

Correct (C++11/Rust/Go): Use atomic operations with acquire/release semantics. In C++, replace plain variables with std::atomic.

// Thread A
shared_data.store(42, std::memory_order_relaxed);
ready.store(true, std::memory_order_release); // Ensures shared_data is visible

// Thread B
if (ready.load(std::memory_order_acquire)) { // Ensures subsequent loads see latest data
    print(shared_data.load(std::memory_order_relaxed));
}

Scenario: Lock-Free Logic

If you are implementing a custom lock-free structure, ensure you are using the language's memory model correctly. If using inline assembly, you must insert a DMB ISH (Data Memory Barrier, Inner Shareable) to ensure all previous memory accesses are completed before proceeding.

Verification and Rollback

Verification:

• Run the TSan build; it must report zero races in the affected module.
• Verify that the STLR/LDAR instructions appear in the binary for the specific synchronization variables.
• Run the stress test on at least two different ARM64 implementations (e.g., one Neoverse-based server and one Apple M-series chip) to ensure the fix is robust across different microarchitectures.

Rollback: Since these fixes involve changing variable types to atomics or adding barriers, rollback consists of reverting the commit to the previous stable x86-compatible state. Note that reverting will re-introduce the instability on ARM64.

Escalation Criteria

Escalate to platform vendors or kernel engineers if:

• Crashes persist despite a clean TSan run and verified LDAR/STLR usage.
• The issue only occurs during Memory-Mapped I/O (MMIO) operations, suggesting a need for DSB (Data Synchronization Barrier) or ISB (Instruction Synchronization Barrier).
• The bug disappears when using a single core (taskset -c 0) but appears on any multi-core configuration despite correct atomics, which may indicate a hardware erratum.