GPUniBench

A unified GPU–GPU communication benchmark for CUDA-aware MPI and NCCL

Features:

• Benchmarks GPU–GPU collectives with both NCCL and CUDA-aware MPI
• Supports common collectives: Allreduce, Bcast, Allgather, Alltoall, Reduce, ReduceScatter, and p2p
• Runs multiple operations in a single pass with different message sizes and datatypes
• Supports both iteration-based and time-based benchmarking
• Produces results as a formatted table or CSV for post-processing

A sample output on 2 GPUs:

Initial Build Steps

git clone https://github.com/MajidSalimi/GPUniBench.git
cd GPUniBench
git submodule init
git submodule update
mkdir build
cd build

Requirements

1. OpenMPI (preferably ≥ 4.1.x)
   • It has to be built with CUDA-awareness to support CUDA-aware MPI benchmarks
2. CUDA (preferably ≥ 11.8.x)
3. NCCL (preferably ≥ 2.19.x)
4. CMake (≥ 3.10)

Example: Modules required on Leonardo Supercomputer

module load nccl/2.19.3-1--gcc--12.2.0-cuda-12.1   cuda/12.1   openmpi/4.1.6--gcc--12.2.0

Build Instructions

cmake ..  -DNCCL_INCLUDE_DIR=$NCCL_INCLUDE  -DNCCL_LIB_DIR=$NCCL_LIB

To build the code:

make -j

Usage and Command-line Arguments

The benchmark is launched using:

mpirun -n <ranks> ./build/gpunibench [OPTIONS]

or:

srun -N <Nnodes> --ntasks-per-node=<NGPUs-per-node> ./build/gpunibench [OPTIONS]

Flag	Description
-h, --help	Help message
-w, --warmups <INT>	Number of warmup iterations
-W, --warmup-time <FLOAT>	Warmup time in seconds
-i, --iter <INT>	Number of iterations
-T, --time <FLOAT>	Benchmarking time per operation in seconds
-s, --opsize <String>*	Operations, message sizes (Bytes), optional datatypes*
-t, --type <Type>**	Global datatype for all message sizes
-B, --blocking	Synchronize after each iteration
-c, --csv	Print CSV output
-d, --details	Print the times of all iterations (must be used with -B)

* For (-s, --opsize <String>): Each -s entry has the form: OPERATION1:SIZE1[:TYPE1],SIZE2[:TYPE2] ... and each operation can handle different message sizes (in Bytes) with different data types. If you wish to have a global type for all the operations and message sizes, use -t instead, and omit the data types from the command.

** If you provide -t, you must not use SIZE:TYPE in -s, and vice versa.

Example 1: Global data type.

mpirun -n 2 ./gpunibench \
    -w 20 \
    -i 100 \
    -t MPI_INT \
    -s NCCL_Allreduce:256,32768  NCCL_Bcast:1024,2048 \
    -B

Example 2: Per-size data types.

mpirun -n 2 ./gpunibench \
    -W 1 \
    -T 3 \
    -s NCCL_Allreduce:256:MPI_INT,32768:MPI_DOUBLE  NCCL_Bcast:1024:MPI_FLOAT,2048:ncclFloat \
    -B

Supported Collective Operations

NCCL

NCCL_Allreduce
NCCL_Alltoall
NCCL_Bcast
NCCL_Allgather
NCCL_Reduce
NCCL_ReduceScatter
NCCL_p2p

CUDA-aware MPI

CUDA_MPI_Allreduce
CUDA_MPI_Alltoall
CUDA_MPI_Bcast
CUDA_MPI_Allgather
CUDA_MPI_Reduce
CUDA_MPI_ReduceScatter
CUDA_MPI_Scatter
CUDA_MPI_p2p

MPI

MPI_Allreduce
MPI_Alltoall
MPI_Bcast
MPI_Allgather
MPI_Reduce
MPI_ReduceScatter
MPI_p2p

Supported Data Types

NCCL Data Types

These data types can only be used with NCCL operations.

For more information about NCCL datatypes, see NCCL Documentation on NVIDIA's website.

ncclInt8
ncclUint8
ncclInt32
ncclUint32
ncclInt64
ncclUint64
ncclFloat16
ncclFloat32
ncclFloat64
ncclBfloat16
ncclChar
ncclInt
ncclHalf
ncclFloat
ncclDouble

MPI Data Types

These data types can be used with both NCCL and CUDA-aware MPI calls. For NCCL, they're mapped to the corresponding NCCL datatype.

MPI_BYTE
MPI_CHAR
MPI_INT
MPI_FLOAT
MPI_DOUBLE

Notes

** Sample scripts for different clusters, e.g. Leonardo, MN5, and LUMI, are in the scripts folder

License

GPUCommBench is released under the MIT License. See the LICENSE file for details.

Disclaimer

This benchmark is provided as is, with no warranty of any kind.
Use it entirely at your own responsability: I’m not responsible for any damage, data loss, wrong conclusions, or other issues that may result from running or relying on it.

Any performance numbers you obtain are not official and are not endorsed or validated by any vendor.