Code crash on Aurora with SCS particle structure; CabM or DPS with >20 million particles per GPU

[zhangchonglin]

Running Comet code on Aurora with different particle structures, here are the observations:

• When using SCS particle structure, the code crashed at the beginning of the simulation. Since chunk_widths is present in the log file, I suspect it crashed during SCS particle structure construction.

NUM_OF_NODES= 1 TOTAL_NUM_RANKS= 12 RANKS_PER_NODE= 12 THREADS_PER_RANK= 1         
Successfully created directory for writing field results on rank 0              
Successfully created directory for writing field results on rank 0              
Successfully created directory for writing field results on rank 0              
Successfully created directory for writing field results on rank 0              
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
terminate called after throwing an instance of 'std::runtime_error'             
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
  what():  Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
x4311c0s3b0n0.hsn.cm.aurora.alcf.anl.gov: rank 2 died from signal 6 and dumped core

• when using CabM or DPS particle structure, the code is running fine if there are less than ~20 million particles per GPU, but crashed if each GPU has more than ~20 million particles:

Segmentation fault from GPU at 0xff00003aa2a20000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 2 (PDP), access: 1 (Write), banned: 1, aborting. 
Abort was called at 274 line in file:                                           
/home/ubit/rpmbuild/BUILD/intel-compute-runtime-25.05.32567.18/shared/source/os_interface/linux/drm_neo.cpp 
Segmentation fault from GPU at 0xff00003aa2a20000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 2 (PDP), access: 1 (Write), banned: 1, aborting. 
x4502c7s4b0n0.hsn.cm.aurora.alcf.anl.gov: rank 6 died from signal 6 and dumped core 
x4502c7s4b0n0.hsn.cm.aurora.alcf.anl.gov: rank 1 died from signal 15     

Code versions used:

- kokkos 4.6.02: https://github.com/kokkos/kokkos
- omega_h master branch: https://github.com/SCOREC/omega_h, at commit: cbb705a
- EnGPar master branch: https://github.com/SCOREC/EnGPar, at commit: 0dc778f
- Cabana master branch: https://github.com/ECP-copa/Cabana, at commit: d9900ea
- PUMIPic `cz/search_3d_plus_yus_aurora-support` branch: https://github.com/SCOREC/pumi-pic, at commit: 1f654d8.
- Comet `cz/aurora` branch: https://github.com/ComputationalGasDynamicsLab/Comet.git, at commit: 38711a4

Modules used to build:

Currently Loaded Modules:
  1) gcc-runtime/13.3.0-ghotoln    (H)  11) mpich/opt/develop-git.6037a7a      21) libmd/1.0.4-q6tzwyj   (H)
  2) gmp/6.3.0-mtokfaw             (H)  12) libfabric/1.22.0                   22) libbsd/0.12.2-wxndujc (H)
  3) mpfr/4.2.1-gkcdl5w            (H)  13) cray-pals/1.4.0                    23) expat/2.6.4-7j6nhb6   (H)
  4) mpc/1.3.1-rdrlvsl             (H)  14) cray-libpals/1.4.0                 24) gdbm/1.23
  5) gcc/13.3.0                         15) bzip2/1.0.8                        25) python/3.10.14
  6) oneapi/release/2025.0.5            16) lz4/1.10.0                         26) gdb/15.2
  7) libiconv/1.17-jjpb4sl         (H)  17) libarchive/3.7.6-brn5xb5      (H)  27) gmake/4.4.1
  8) libxml2/2.13.5                     18) libmicrohttpd/0.9.50-by7j3p7  (H)  28) cmake/3.30.5
  9) hwloc/2.11.3-mpich-level-zero      19) sqlite/3.46.0-w5wc5lh         (H)
 10) yaksa/0.3-7ks5f26             (H)  20) elfutils/0.186-edvhjaw        (H)

[zhangchonglin]

CabM and DPS particle structure crashes may be related to earlier issue: #119.

[Sichao25]

Hi, thank you for reporting these bugs. Based on my testing, I've confirmed that the CabM and DPS crashes are indeed related to the earlier issue you mentioned. However, I was unable to reproduce the SCS crash in the PUMIPic code.

Did you encounter this crash during the same simulation step described in issue 119? If not, would you mind sharing the relevant part of the Comet code or providing a brief description if it's not confidential?

[zhangchonglin]

@Sichao25: Hi Sichao, thank you for looking into this. I have invited you to join my group GitHub. Once you join, I will add you to the Comet code repository. Thanks!

[Sichao25]

@zhangchonglin Thank you for adding me to the group. I have a few questions: In your testing so far, does this issue occur only on Aurora, or does it happen on other machines as well? Is comet.cpp the simulation that's causing the crash, or is it a different test? If you have a minimal reproducer, that would also be very helpful.

[zhangchonglin]

@Sichao25: Hi Sichao, thank you for looking into this further:

• This is only happening on Aurora. It was running fine on other machines when using SCS structure with any number of particles per GPUs.
• On Aurora, it crashed at the beginning of the simulation with SCS particle structure.
• You can use this test case , just supply Comet binary file in the same folder, and copy the run scripts to the root folder of this case when submitting testing jobs.

[jacobmerson]

@zhangchonglin can you see if you are able to create a minimum reproducer that is just PUMI-PIC? That will help us track it down more effectively and then we can include it into our testing strategy once we identify the bug.

Thanks!

[zhangchonglin]

@jacobmerson: good suggestion. I will try to test this with the PseudoXGCm test once Aurora is back online again.

[jacobmerson]

Great thank you!