Add NVTX marker instrumentation #2345
Conversation
edopao
left a comment
edopao
left a comment
There was a problem hiding this comment.
Definitely useful, in my opinion. Adding @egparedes because this PR is useful in the tracing of asynchronous execution to visualize the program call.
src/gt4py/next/program_processors/runners/dace/workflow/translation.py
Outdated
Show resolved
Hide resolved
| dace.Memlet(f"{output}[0]"), | ||
| ) | ||
|
|
||
| if (config.COLLECT_METRICS_LEVEL == metrics.GPU_TX_MARKERS) and gpu: |
There was a problem hiding this comment.
Your current check is at compile time. The check in the instrumentation above is at runtime. I propose to always add NVTX instrumentation, because the config.COLLECT_METRICS_LEVEL value is meant to be a runtime setting. It is still possible to skip SDFG instrumentation by setting use_metrics=False on the dace backend, at compile-time.
There was a problem hiding this comment.
I am not sure I would be up to enabling NVTX ranges by default since they will add some (really minimal) overhead to the CPU side. If use_metrics=False can be set by default then it would be fine but it would increase some more complexity for production and profiling runs
|
Thanks @iomaganaris for this PR and @edopao for pointing me here. I would like to make this PR larger to add NVTX markers in the whole program/operator hot path, not only in the DaCe backend. I didn't have time today but I propose to discuss offline the scope and design of this together, since @edopao has also done some work with this. |
edopao
dismissed
their stale review
LGTM but Enrique had some ideas to extend this PR.
3 participants
Not sure if this is useful but I'm opening a PR to avoid losing these changes.
It generates the following NVTX ranges:
Requirements