nvidia-smi doesn't reflect the nvidia-cuda installation

[ZhiliangWu]

Dear community,

I am using the nvidia-cuda feature based on mcr.microsoft.com/devcontainers/base:jammy.

{
  "name": "GenX v2.0 reference development environment. Supported languages: Python",
  "image": "mcr.microsoft.com/devcontainers/base:jammy",
  "runArgs": ["--gpus", "all"], 
  "features": {
    "ghcr.io/devcontainers/features/nvidia-cuda": {
      "installCudnn": true,
      "cudaVersion": "12.4"
    }
}
  },

On the host machine, we have already nvidia driver installed and the information from nvidia-smi shows as following

$ nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|

However, inside the devcontainer, I cannot see any cudaversion I selected being reflected. I.e., I expected the information above to be something like 12.4 but what I got is

$ nvidia-smi    
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|

When I check the installation inside of the devcontainer, I do see cuda get installed

$ dpkg -l | grep cuda
ii  cuda-cudart-12-4                12.4.127-1                              amd64        CUDA Runtime native Libraries
ii  cuda-keyring                    1.0-1                                   all          GPG keyring for the CUDA repository
ii  cuda-libraries-12-4             12.4.1-1                                amd64        CUDA Libraries 12.4 meta-package
ii  cuda-nvrtc-12-4                 12.4.127-1                              amd64        NVRTC native runtime libraries
ii  cuda-opencl-12-4                12.4.127-1                              amd64        CUDA OpenCL native Libraries
ii  cuda-toolkit-12-4-config-common 12.4.127-1                              all          Common config package for CUDA Toolkit 12.4.
ii  cuda-toolkit-12-config-common   12.8.90-1                               all          Common config package for CUDA Toolkit 12.
ii  cuda-toolkit-config-common      12.8.90-1                               all          Common config package for CUDA Toolkit.
ii  libcudnn9-cuda-12               9.8.0.87-1                              amd64        cuDNN runtime libraries for CUDA 12.8

any suggestion on how to make the cuda version inside of the devcontianer match the one I configured for the nvidia-cuda feature?

[ZhiliangWu]

The same if I enable all boolean options as

    "ghcr.io/devcontainers/features/nvidia-cuda": {
      "installCudnn": true,
      "installCudnnDev": true,
      "installNvtx": true,
      "installToolkit": true,
      "cudaVersion": "12.4"
    }

[Kaniska244]

Hello @ZhiliangWu ,

Thanks for reporting this issue. This behavior is probably specific to nvidia-smi tool. Kindly check if this discussion is helpful for your scenario.

Below snippet I see from the discussion.

Similarly, when using docker, the nvidia-smi command will generally report the driver version installed on the base machine, whereas other version methods like nvcc --version will report the CUDA version installed inside the docker container.

With Regards,
Kaniska