Intermittent deadlock in ompi_sync_wait_mt on ARM64 (Graviton)

[blkqi]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

• v4.1.6
• v5.0.6

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

• Ubuntu distribution
• AWS distribution (EFA)

Please describe the system on which you are running

• Operating system/version: Amazon Linux 2023, Ubuntu 24
• Computer hardware: c7gn.16xlarge (and other Graviton2 or Graviton3 instance types)
• Network type: ENI (TCP)

---

Summary

We're seeing intermittent deadlocks in multi-threaded MPI applications on AWS Graviton that never occur on x86_64. All threads end up stuck in ompi_sync_wait_mt(). After analyzing the code, I believe this is due to missing memory barriers in the ARM64 atomic operations used by the wait_sync mechanism.

Environment

• Open MPI: v5.0.8 (also tested v4.1.6, same issue)
• Platform: AWS Graviton2 or Graviton3 (ARM64/aarch64)
• Compiler: GCC -O2
• Application: Multi-threaded MPI with concurrent operations from multiple threads

What we observe

When the deadlock occurs, backtraces show all ranks stuck in ompi_sync_wait_mt(). Some threads are spinning in opal_progress(), others are blocked on condition variables. The affected MPI operations include MPI_Ssend, MPI_Mprobe, and MPI_Waitall.

Unfortunately there is no known minimal reproducer that triggers the deadlock deterministically. The issue appears intermittently at production-scale workloads with high thread contention. It never happens on x86_64 with the same code.

Analysis

Looking at the code, ompi_sync_wait_mt() implements a counting semaphore pattern:

The completion thread decrements sync->count (wait_sync.h:144):

if (0 != (OPAL_THREAD_ADD_FETCH32(&sync->count, -updates))) {
    return;
}
WAIT_SYNC_SIGNAL(sync);

The waiting thread polls it (wait_sync.c:118-120):

OPAL_THREAD_ADD_FETCH32(&num_thread_in_progress, 1);
while (sync->count > 0) {
    opal_progress();
}

The problem is in opal/include/opal/sys/arm64/atomic.h. The OPAL_ASM_MAKE_ATOMIC macro (lines 286-295) uses ldxr/stxr which are atomic but lack acquire/release semantics. Additionally, the while (sync->count > 0) is a plain C read with no acquire semantics.

On ARM64's weak memory model, this means:

• The completion thread's decrement might not be visible to the waiting thread
• The waiting thread might read a stale cached value

Interestingly, the same file has opal_atomic_swap_* (lines 247-258) which correctly uses ldaxr/stlxr with acquire/release semantics. So there's an inconsistency.

History

This appears to have been introduced in commit 7893248 (Nov 2017) when fetch-and-op atomics were added. It affects all v4.x and v5.x releases. There was a partial fix attempt in commit 5e13f02 (Jan 2021) that added atomic_llsc.h with proper barriers, but the OPAL_ASM_MAKE_ATOMIC macro wasn't updated.

Proposed fix

I think both of these changes are needed:

1. Update OPAL_ASM_MAKE_ATOMIC to use ldaxr/stlxr (matching opal_atomic_swap_*)
2. Change the plain read to use an atomic load with acquire semantics

References

• Commit 7893248 (introduced the issue)
• Commit 5e13f02 (partial fix attempt)

[blkqi]

Hey @bwbarrett - given your work in HPC/MPI at AWS, I wanted to flag this Graviton deadlock issue specifically. We are hitting this in production on both i4g and c7gn instance types. Happy to assist however I can.

[darrylabbate]

Tagging ourselves here - @open-mpi/efa

[bosilca]

I haven't looked at this code in a while, but the count is marked as atomic, and its usages are across a function call, so there should be no caching. This is something you can confirm with a debugger, looked at the register and at the memory location of the variable.

But before going deeper into this, why is your build using the atomic operations instead of using the compiler atomic support (which should be the default if available) ? Can you provide the config.log or at least the atomic detection part.

[darrylabbate]

Thanks for the detailed write-up. I checked this against current main, and the code paths you pointed at still look materially the same. I agree there may be a real synchronization bug here in wait_sync; with ompi_wait_sync_t.count being an atomic type, and the wait side polling it without an explicit acquire load, while the ordering contract with the completion side is not completely obvious.

Update OPAL_ASM_MAKE_ATOMIC to use ldaxr/stlxr (matching opal_atomic_swap_*)

First, is your OMPI build actually utilizing the ASM backend? By default, OMPI prefers GCC builtins, then C11 atomics; falling back to the hardcoded ASM backend only if those are unavailable or disabled. Otherwise, I wouldn't expect changes to OPAL_ASM_MAKE_ATOMIC to affect any behavior you observe.

Second, I’m not necessarily convinced that OPAL_ASM_MAKE_ATOMIC should be changed just to match opal_atomic_swap_*. The preferred backends currently treat these operations as relaxed, so I would be hesitant to strengthen the asm fallback semantics based solely on the fact that opal_atomic_swap_* is stronger in that implementation.

Change the plain read to use an atomic load with acquire semantics

Agreed. I’d be more inclined to make the ordering requirements explicit in wait_sync first, rather than changing the semantics of the general-purpose ARM64 fetch-op atomics for everyone.

[blkqi]

You're right that my build is not using the ASM backend — OPAL_USE_ASM_ATOMICS 0 in my config.log — so the OPAL_ASM_MAKE_ATOMIC change is irrelevant here and I agree it shouldn't be generalized.

I agree with your preference to make the ordering requirements explicit in wait_sync directly, and your point about the ordering contract being non-obvious is well taken. Looking more closely at the code, I think the missing acquire semantics on sync->count may be more significant than I initially thought — particularly around the check after cond_wait returns. I'll update my patch accordingly and report back.

[blkqi]

Following up with a fix and some findings from production.

@darrylabbate — per your suggestion, we made the ordering requirements explicit in wait_sync first. PR #13791 adds opal_atomic_wmb() before the CAS that publishes the sync pointer in ompi_request_wait_completion, and opal_atomic_rmb() after the swap in ompi_request_complete. Same pattern in wait_any, wait_all, wait_some. This fixed the most common deadlock (lost wakeup in ompi_sync_wait_mt) on ARM64 (observed on Graviton2 through Graviton4).

Fixing the request layer exposed additional ordering and logic bugs in pml/ob1 that we're still working through. Should I file a separate issue for the ob1 work, or would you prefer it tracked here?