Misc async thread fixes - #697
Draft
jayeshkrishna wants to merge 15 commits into
Draft
Conversation
jayeshkrishna
added
Next Release
Comment on lines
+15
to
+21
| int nthreads = 1; | ||
| #if PIO_USE_ASYNC_WR_THREAD | ||
| tids_ = PIO_Util::PIO_async_tpool_manager::get_tpool_instance()->get_thread_ids(); | ||
| nthreads += PIO_Util::PIO_async_tpool_manager::get_num_threads(); | ||
| if(io_comm != MPI_COMM_NULL){ | ||
| tids_ = PIO_Util::PIO_async_tpool_manager::get_tpool_instance()->get_thread_ids(); | ||
| assert(static_cast<int>(tids_.size()) == nthreads - 1); | ||
| } |
Comment on lines
+136
to
+142
| int nthreads = 1; | ||
| #if PIO_USE_ASYNC_WR_THREAD | ||
| /* Number of threads including the main thread */ | ||
| std::size_t nthreads = tids_.size() + 1; | ||
| assert(union_comms_.size() == nthreads); | ||
| for(std::size_t tidx = 1; tidx < nthreads; tidx++){ | ||
| nthreads += PIO_Util::PIO_async_tpool_manager::get_num_threads(); | ||
| #endif | ||
| assert(static_cast<int>(union_comms_.size()) == nthreads); | ||
| for(int tidx = 1; tidx < nthreads; tidx++){ |
jayeshkrishna
force-pushed
the
jayeshkrishna/fix_num_threads
branch
from
June 7, 2026 00:47
b7c610b to
5f4ae01
Compare
Add interface to the thread pool manager to get the number of threads (configured for the thread pool)
The compute processes do not require any threads. However these processes need to know the number of threads (configured in the thread pool - in the I/O procs) On I/O processes use the thread pool manager to init the thread pool and get the thread ids. On compute procs use the thread pool manager to just get the number of threads.
Disable waiting on pending HDF5 ops when closing non-HDF5 files
Disable waits for HDF5 ops when opening files
Avoid duping comms that the I/O threads don't use (or should be using)
Re-formatting code in pio_err function This commit does not change the functionality (or add new code)
Since threads no longer have valid global comms, updating the logic to print error by checking for validity of the comms before using it
Threads should not be using global comms. Updating barrier in async op to be on thread-specific I/O comm
When closing file ensure that we don't sync on the async I/O threads
Since not all error handlers are multi-threaded use the return error handler when closing files (in async+threaded case).
The new test case creates many files and writes to them using one decomp per file. The test case has two phases for each I/O type, 1) Create decomposition, create file and write to file 2) Sync data (close+open for sync if needed), read and verify, close+delete files, free decomposition Phase 1 is executed for all files before moving to phase 2
Adding a simple stdout logger for tracing the async I/O ops (logs are commented/disabled by default) Once the tracer/logger is multi-threaded we can get rid of these loggers
Adding a new configure option to choose whether to trace I/O decompositions (while tracing APIs). By default its turned off
Since not all MPI libraries support concurrent operations (from multiple threads) on multiple files ensure that we wait on pending async ops before proceeding with a synchronous operation (read of a different file, sync op using a different iotype etc) This is not efficient and we need to improve read perf in a future PR.
Adding missing static for function level mutex when creating thread-specific comms.
jayeshkrishna
force-pushed
the
jayeshkrishna/fix_num_threads
branch
from
July 21, 2026 16:33
e68c2c1 to
2fb5031
Compare
jayeshkrishna
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The number of threads were determined by first
getting the thread pool instance and querying it.
However getting the thread pool instance involves
creating the threads, so threads were created for
all processes.
We now use a static method for querying the
number of threads and the thread pool is
initialized only for I/O processes.
Also, removing waits on pending hdf5 operations
when closing/opening files