Cannot use GPU on CST in Poly-Grames node006/node007

[jcohenadad]

@4rnaudB in addition to documenting your debugging, can you pls also describe a way to test a simulation (eg: open a 'dummy' project in CST, click on solver, etc.)
--> Presentation on how to set up the GPU acceleration

Relevant documentation: https://updates.cst.com/downloads/GPU_Computing_Guide_2022.pdf

[JustinDeM]

The GPU on Node006 and Node007 is a Tesla V100S-PCIE-32GB :

which is not directly supported like the Tesla V100-PCIE-32GB. We will have to the following (from https://updates.cst.com/downloads/GPU_Computing_Guide_2022.pdf) :

[JustinDeM]

I couldn't manage to find whether the Tesla-V100S is capable of running CUDA 11.2 code, but I suspect it is because it is more powerful than the Tesla-V100. If so, to enable the GPU in CST, administrator privilege would be required to set environment variables.

[jcohenadad]

administrator privilege would be required to set environment variables

so we need to loop in JS-- i'll take care of it

EDIT: email sent-- waiting for an answer

[jcohenadad]

testing on node006, I tried launching HWAccDiagnostics, but got this:

Likely related to #61 (comment)

[jcohenadad]

@JustinDeM @4rnaudB did you manage to run HWAccDiagnostics? Have you tried installing CUDA drivers?

[4rnaudB]

trying it right now

[JustinDeM]

We get this error message :

[4rnaudB]

Seems we don't have access to C:\ we can't run it. There is another test you can do if you have access to see if the GPU is recognized.

[4rnaudB]

Also, we got an error this morning with node006 saying no GPU was recognized but we don't get the same error with node007 when running the simulation.

[JustinDeM]

When we run the Check GPU Computing macro on Node007, we get this :

(The Tesla-V100s is a Volta series).

[jcohenadad]

There is another test you can do if you have access to see if the GPU is recognized.

I don't have access-- hopefully a Poly admin will help us ASAP

[jcohenadad]

When we run the Check GPU Computing macro on Node007, we get this :

Ah! That's interesting, when I ran it on node006 I got the error message reported in #61 (comment)

[jcohenadad]

#61 (comment) is pretty encouraging because the GPU is recognized! What is ECC? can it be disabled?

EDIT: https://www.nvidia.com/content/Control-Panel-Help/vLatest/en-us/mergedProjects/nvwks/To_change_the_ecc_state.htm

[JustinDeM]

Error Correction Code. We are trying to change it but the Nvidia Control Panel won't open. It could be related to the fact that someone else is using a lot of the ressources of Node007 right now :

[jcohenadad]

It could be related to the fact that someone else is using a lot of the ressources of Node007 right now :

Good point-- we should ask the GRAMES admin if they have a way to allocate ressource via a priority list, as done in Compute Canada with SLURM. --> #63

[jcohenadad]

Answer from JS:

Ce serveur vient d’être monté il n’y a pas ci longtemps. J’ai vu le message sur GitHub, je vais ajouter la variable un peu plus tard… Pour le moment seulement ADS été utilisé sur ces serveurs

[4rnaudB]

@4rnaudB in addition to documenting your debugging, can you pls also describe a way to test a simulation (eg: open a 'dummy' project in CST, click on solver, etc.)

Relevant documentation: https://updates.cst.com/downloads/GPU_Computing_Guide_2022.pdf

Should this be an issue or a powerpoint? (I'm planning to put images for a step by step process)

[jcohenadad]

Should this be an issue or a powerpoint? (I'm planning to put images for a step by step process)

hum, good question. If you prefer creating a PPTX, then I suggest you do it with Gslide, put the document here and link it in this issue?

[4rnaudB]

Sounds good

[4rnaudB]

The problem is that the link to the google slide would be at the end of the issue so it might not be very intuitive/practical to add it in this issue.

[jcohenadad]

The problem is that the link to the google slide would be at the end of the issue so it might not be very intuitive/practical to add it in this issue.

so how about adding it at the top? #61 (comment)

[4rnaudB]

Oh nice, didn't know I could edit your comment.