Skip to content

Debug hipergator mpi #1089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
danieljvickers wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Debug hipergator mpi #1089

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3b24b31
works on 2 ranks with GPU
Dec 15, 2025
45bdac5
Added more writes
Dec 15, 2025
9d8edcc
Updated modules:
Dec 15, 2025
fa1a187
Merge branch 'master' into debug-hipergator-mpi
sbryngelson Dec 15, 2025
4e57fac
Added python requirement
Dec 15, 2025
a0b3f00
Merge branch 'debug-hipergator-mpi' of github.com:danieljvickers/MFC-…
Dec 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion src/post_process/m_data_input.f90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,6 @@ impure subroutine s_read_serial_data_files(t_step)
STATUS='old', ACTION='read')
read (1) q_cons_vf(i)%sf(0:m, 0:n, 0:p)
close (1)
print *, q_cons_vf(i)%sf(:, 0, 0)
else
call s_mpi_abort('File q_cons_vf'//trim(file_num)// &
'.dat is missing in '//trim(t_step_dir)// &
Expand Down
7 changes: 4 additions & 3 deletions src/pre_process/m_data_output.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -765,7 +765,7 @@ contains

call MPI_FILE_SET_VIEW(ifile, disp, mpi_io_p, MPI_IO_DATA%view(i), &
'native', mpi_info_int, ierr)
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
Comment on lines +768 to 769
Copy link
Contributor

@qodo-code-review qodo-code-review bot Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: In the MPI_FILE_WRITE_ALL call, correct the count parameter from data_size*mpi_io_type to data_size to prevent writing an incorrect amount of data and ensure consistency with other changes in the PR. [possible issue, importance: 9]

Suggested change
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size, &
mpi_io_p, status, ierr)

end do
!Additional variables pb and mv for non-polytropic qbmm
Expand All @@ -778,7 +778,7 @@ contains

call MPI_FILE_SET_VIEW(ifile, disp, mpi_io_p, MPI_IO_DATA%view(i), &
'native', mpi_info_int, ierr)
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
end do
end if
Expand All @@ -792,7 +792,8 @@ contains

call MPI_FILE_SET_VIEW(ifile, disp, mpi_io_p, MPI_IO_DATA%view(i), &
'native', mpi_info_int, ierr)
call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &

call MPI_FILE_WRITE_ALL(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size, &
mpi_io_p, status, ierr)
end do

Expand Down
11 changes: 6 additions & 5 deletions src/simulation/m_start_up.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -679,15 +679,15 @@ contains
do i = 1, sys_size!adv_idx%end
var_MOK = int(i, MPI_OFFSET_KIND)

call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
Comment on lines +682 to 683
Copy link
Contributor

@qodo-code-review qodo-code-review bot Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: In the MPI_FILE_READ call, correct the count parameter from data_size*mpi_io_type to data_size to prevent reading an incorrect amount of data and ensure consistency with other changes in the PR. [possible issue, importance: 9]

Suggested change
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size, &
mpi_io_p, status, ierr)

end do
!Read pb and mv for non-polytropic qbmm
if (qbmm .and. .not. polytropic) then
do i = sys_size + 1, sys_size + 2*nb*nnode
var_MOK = int(i, MPI_OFFSET_KIND)

call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
end do
end if
Expand Down Expand Up @@ -824,7 +824,7 @@ contains

call MPI_FILE_SET_VIEW(ifile, disp, mpi_io_p, MPI_IO_DATA%view(i), &
'native', mpi_info_int, ierr)
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
end do
!Read pb and mv for non-polytropic qbmm
Expand All @@ -836,7 +836,7 @@ contains

call MPI_FILE_SET_VIEW(ifile, disp, mpi_io_p, MPI_IO_DATA%view(i), &
'native', mpi_info_int, ierr)
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &
call MPI_FILE_READ(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size*mpi_io_type, &
mpi_io_p, status, ierr)
end do
end if
Expand All @@ -849,7 +849,8 @@ contains

call MPI_FILE_SET_VIEW(ifile, disp, mpi_io_p, MPI_IO_DATA%view(i), &
'native', mpi_info_int, ierr)
call MPI_FILE_READ_ALL(ifile, MPI_IO_DATA%var(i)%sf, data_size*mpi_io_type, &

call MPI_FILE_READ_ALL(ifile, MPI_IO_DATA%var(i)%sf(1, 1, 1), data_size, &
mpi_io_p, status, ierr)
end do
end if
Expand Down
12 changes: 5 additions & 7 deletions toolchain/modules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,8 @@ san-gpu nvhpc cuda cray-mpich

h hipergator
h-gpu nvhpc/25.9
h-gpu CUDA_HOME="/apps/compilers/cuda/12.8.1"
h-all HPC_OMPI_DIR="/apps/mpi/cuda/12.8.1/nvhpc/25.3/openmpi/5.0.7"
h-all HPC_OMPI_BIN="/apps/mpi/cuda/12.8.1/nvhpc/25.3/openmpi/5.0.7/bin"
h-all OMPI_MCA_pml=ob1 OMPI_MCA_coll_hcoll_enable=0
h-gpu PATH="/apps/mpi/cuda/12.8.1/nvhpc/25.3/openmpi/5.0.7/bin:${PATH}"
h-all LD_LIBRARY_PATH=/apps/compilers/cuda/12.8.1/lib64:$LD_LIBRARY_PATH
h-gpu MFC_CUDA_CC=100 NVHPC_CUDA_HOME="/apps/compilers/cuda/12.8.1"
h-gpu openmpi/5.0.7
h-gpu python/3.12
h-gpu MFC_CUDA_CC=100
h-all LD_LIBRARY_PATH=/apps/compilers/nvhpc/25.9/Linux_x86_64/25.9/math_libs/12.9/lib64:$LD_LIBRARY_PATH
h-all UCX_NET_DEVICES="mlx5_4:1,mlx5_7:1,mlx5_8:1,mlx5_9:1,mlx5_10:1,mlx5_13:1,mlx5_14:1,mlx5_15:1"
Loading