Script to probe the nccl libraries that PyTorch uses #267
Conversation
verdimrc
force-pushed
the
probe-pt-dlopen-nccl-aws-ofi-nccl
branch
from
cf57d36
to
07a509c
Compare
|
cancel or do we move forward with it? |
verdimrc
changed the title
verdimrc
force-pushed
the
probe-pt-dlopen-nccl-aws-ofi-nccl
branch
from
b6f4189
to
6463723
Compare
|
Ready for review. |
| The scripts in this folder disambiguate the exact NCCL libraries that a PyTorch application actually | ||
| uses, in the presence of potentially multiple installed versions. | ||
|
|
||
| ## 1. Motivation |
There was a problem hiding this comment.
Include usage at the top
| By now, hopefully you're convinced on the need for a runtime probe to pinpoint the version of NCCL | ||
| loaded by PyTorch. | ||
|
|
||
| ## 2. Howto |
| Via Slurm: `srun -l -N1 ./probe-pt-nccl-aws-libs.sh` | ||
|
|
||
| ```console | ||
| $ srun -l -N1 ./probe-pt-nccl-aws-libs.sh |
There was a problem hiding this comment.
Can this be used as a library within another script? i.e. I want to use it in my efa-versions.py to grab the nccl version but invoking the shell script and grep-ing the output is less than ideal.
| 0: cat /opt/amazon/efa_installed_packages: | ||
| 0: # EFA installer version: 1.30.0 | ||
| # Debug packages installed: no | ||
| 0: # Packages installed: |
|
|
||
| echo | ||
| echo "cat /opt/amazon/efa_installed_packages:" | ||
| cat /opt/amazon/efa_installed_packages |
my quick 2c: for this purpose, it's best to re-implement the logic in pure python. Because the bulk of the work is done by Plus, with Python you'll have access to proper representation of the structured object, and I think this is a better way to have a machine-readable version for Python land. My shell script is meant for human-readability, and in it's current form is far away for the automation purpose you might have in mind. |
KeitaW
force-pushed
the
probe-pt-dlopen-nccl-aws-ofi-nccl
branch
from
361d6ba
to
16db56b
Compare
KeitaW
force-pushed
the
probe-pt-dlopen-nccl-aws-ofi-nccl
branch
from
16db56b
to
e815903
Compare
KeitaW
force-pushed
the
main
branch
2 times, most recently
from
44e448e
to
1209815
Compare
|
Closing as it's stale. |
Issue #, if available: close #252
Description of changes: Probe what PyTorch actually uses for the nccl stacks.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.