Enable tracing of the I/O tasks from the thread pool

[kingcrimsontianyu]

KvikIO parallel I/O API pread()/pwrite() uses a thread pool to concurrently process I/O requests. There is a demand for tracing the tasks and visualizing their time ranges in the Nsight System profiler. The requested feature is described below.

• Basic feature: KvikIO shall trace the initial I/O request pread()/pwrite() (run on the main thread and return immediately once the I/O tasks are submitted) and its constituent I/O tasks (run on the worker threads in the background). Tasks originating from the same pread()/pwrite() call shall be uniquely identified (as much as possible), for instance, by (1) receiving a unique NVTX color for the time range and (2) receiving a unique call index as the NVTX payload.

• Good-to-have: KvikIO shall correlate the initial I/O request pread()/pwrite() and its subsequent I/O tasks. They shall be given the same color and same call index. Note that pread()/pwrite() has already been given the initial I/O size as its NVTX payload, necessitating the use of the new binary payload feature in NVTX/Nsight System (which allows for arbitrary user-defined schema). NVTX binary payload is still on the dev branch at the moment.

• Good-to-have: KvikIO shall rename the special-purpose worker threads from the thread pool, so that in the Nsight System UI, they can be easily distinguished from other threads. This can be done using the NVTX custom resource feature. Despite the support on the NVTX side, the custom resource is still not supported by Nsight System at the moment.