CUDA module cannot be loaded

[roy12369]

#Hi @bensloane ,
I applied the python and showed the error. Does it mean that I must have GPU in my viya server?
If so,how can I change to run on the cpu?

[bensloane]

Hey @roy12369 is there any particular reason you are running this as a python script? I would first just try running it as is in Jupyter. Looks like you receive the error when you start training the model and by default in this notebook i have gpu=dict(devices={0,1}) in the call to dltrain. I would remove this line and see if that helps at all. Thanks.

[roy12369]

Hi @bensloane ,
Would it cause any different result when running it as python script or Jupyter? If so,I would advice my customer to run the code on the Jupyter. Looks like the issue somehow relates to the GPU.

[bensloane]

No it would not but the notebook was built to be interactive and useful for exploring the fashion mnist data. You just lose a little of that investigative workflow outside of Jupyter but of course you can still run this as a script more just curious as to why. Yes just remove that line stated above and you will train on the cpu.

[roy12369]

Hi @bensloane ,
Got it! Many Thanks! I thought my customer just edited the code on spyder and saved it as a python script. Then just tried to send me the error screen with running the python script.

[nadolsw]

Hello, I am a current SAS employee attempting to stand up a demo using the SAS ML CAS Free Trial accessed via this link: https://www.sas.com/en_us/software/machine-learning-cloud.html

I have encountered the same error listed above (ERROR: CUDA module cannot be loaded.) and was hoping to get some guidance on how to resolve or who to contact for further assistance as this comment chain is the only reference I can find online related to the error.

Here is the code I am attempting to submit:
74 ods output OptIterHistory=ObjectModeliter;
75 proc cas;
76 dlTrain / table={name='SmallImageDatashuffled', where='PartInd=1'} model='ConVNN'
77 modelWeights={name='ConVTrainedWeights_d', replace=1}
78 bestweights={name='ConVbestweights', replace=1}
79 inputs='image'
80 target='label' nominal={'label'}
81 GPU=True
82 ValidTable={name='SmallImageDatashuffled', where='PartInd=2'}
83 optimizer={minibatchsize=80,
84 algorithm={method='ADAM', lrpolicy='Step', gamma=0.6, stepsize=5
85 beta1=0.9, beta2=0.999, learningrate=.01}
86 maxepochs=25}
87 seed=12345
88 ;
89 run;
NOTE: Active Session now CASAUTO.
ERROR: CUDA module cannot be loaded.
ERROR: The action stopped due to errors.

Note that the code dos run if I change GPU=False but I'm attempting to leverage the GPU rather than CPU for this task. Any help is much appreciated!

[bensloane]

Are you running on a server that has CUDA installed? Using the trial environment does not include a GPU as far as I know. Try changing GPU=False. Also I recommend checking the dlpy repo for more demo examples!

[nadolsw]

Ah, that would explain it. Do you happen to know if there are any commands which could be submitted in order to verify whether or not the cloud server has a GPU available?

[bensloane]

You want to see if CUDA is installed so there's a couple places to check. By no means is this an exhaustive list but you could try the following:

nvidia-smi
nvcc --version

If you don't have luck with those take a look at some of the CUDA docs

[nadolsw]

Wonderful - thank you so much!