GPU Max 1100 doesn't support querying the available free memory

[sharvil10]

Describe the bug
We have a K8s cluster with 2 nodes and 8 GPU Max 1100s each. We installed the GPU plugin v0.34.0 to access the GPU in the cluster. However, when we run this command in the pod python -c "import torch; print(torch.xpu.mem_get_info(torch.xpu.current_device())[1])" it fails with the following error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/vllm/lib64/python3.12/site-packages/torch/xpu/memory.py", line 194, in mem_get_info
    return torch._C._xpu_getMemoryInfo(device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The device (Intel(R) Data Center GPU Max 1100) doesn't support querying the available free memory. You can file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize its implementation.

However, the above command works in a container on the same node. I have attached the pod definition and container command below.

sudo nerdctl run -it \
  --name xpu \
  -e VLLM_LOGGING_LEVEL=DEBUG \
  -e http_proxy=http://proxy-dmz.intel.com:912 \
  -e https_proxy=http://proxy-dmz.intel.com:912 \
  -e no_proxy="localhost,127.0.0.1,.maas,10.0.0.0/8,172.16.0.0/16,192.168.0.0/16,134.134.0.0/16,.maas-internal,.svc" \
  --device /dev/dri \
  --entrypoint="bash" \
  --privileged \
  ghcr.io/llm-d/llm-d-xpu:v0.3.0

To Reproduce

apiVersion: v1
kind: Pod
metadata:
  name: xpu
  namespace: default
spec:
  
  containers:
  - name: xpu
    image: ghcr.io/llm-d/llm-d-xpu:v0.3.0
    imagePullPolicy: Always
    command:
    - bash
    stdin: true
    tty: true
    env:
    - name: VLLM_LOGGING_LEVEL
      value: DEBUG
    - name: http_proxy
      value: http://proxy-dmz.intel.com:912
    - name: https_proxy
      value: http://proxy-dmz.intel.com:912
    - name: no_proxy
      value: "localhost,127.0.0.1,.maas,10.0.0.0/8,172.16.0.0/16,192.168.0.0/16,134.134.0.0/16,.maas-internal,.svc"
    resources:
      limits:
        gpu.intel.com/i915: "8"
      requests:
        gpu.intel.com/i915: "8"
  restartPolicy: Never

This same pod works with privileged: true and removing the i915 request and limits.

Expected behavior
The GPU memory command should work if it works in containers.

System (please complete the following information):

• OS version: [e.g. Ubuntu 22.04]
• Kernel version: [e.g. Linux 5.15]
• Device plugins version: [e.g. v0.34.0]
• Hardware info: [e.g. Intel dGPU Max 1100]

[sharvil10]

@tkatila has provided following solution that works beautifully.

Add a volumeMount to the plugin container:

        - mountPath: /dev/dri/by-path
          name: bypath

And add the corresponding volume to the Pod:

      - emptyDir: {}
        name: bypath

Then remove any CDI files from the hosts:

sudo rm -rf /var/run/cdi/intel.cdi.k8s.io-gpu*

And finally restart the plugin:

kubectl rollout restart daemonset intel-gpu-plugin

[tkatila]

Thanks for reporting this @sharvil10

The underlying problem is with the by-path symlinks under /dev/dri/by-path. GPU plugin mounts them based on the allocated GPUs, but while they point to the same character devices, they are not anymore symlinks. While container runtime mounts them, they become hard links(?) or such. The pytorch stack (or some library below) uses the symlink targets to navigate the sysfs (i.e. /sys/class/drm//) and that then fails.

Dropping the by-path mounts seems to fix it as well as mounting the by-path directory itself. Sadly, I didn't find a way to mount the individual symlinks as symlinks to the container.

[eero-t]

The underlying problem is with the by-path symlinks under /dev/dri/by-path. GPU plugin mounts them based on the allocated GPUs, but while they point to the same character devices, they are not anymore symlinks. While container runtime mounts them, they become hard links(?) or such.

Yes, they are just resolved, i.e. they become the files that they point to. That's what Linux kernel bind mounts do by default.

Semantically mounting single symlink from host into a container as symlink makes less sense than mounting the pointed file there, and there are some other good reasons for this default, see: https://github.com/opencontainers/runc/blob/dbe5acade3d0f84b5c1bd1184b1bbf5a533cdfa1/libcontainer/rootfs_linux.go#L192-L195

The pytorch stack (or some library below) uses the symlink targets to navigate the sysfs (i.e. /sys/class/drm//) and that then fails.

Dropping the by-path mounts seems to fix it as well as mounting the by-path directory itself. Sadly, I didn't find a way to mount the individual symlinks as symlinks to the container.

While OCI spec specifies (non-mandatory) Linux specific nosymfollow option: https://github.com/opencontainers/runtime-spec/blob/main/config.md#linux-mount-options

And at least runc implements support for that: https://github.com/opencontainers/runc/blob/main/libcontainer/rootfs_linux.go#L462

Kubernetes device plugins API (Mount struct) does not include support for specifying that: https://github.com/kubernetes/kubelet/blob/master/pkg/apis/deviceplugin/v1beta1/api.proto#L190

So above 2 options are best that can be done, unless Kubernetes plugin API is improved.

While mapping whole by-path directory to container reveals PCI IDs and device file names of the other GPUs, I don't think that's any kind of security issues. But presence symlinks for the other GPUs (that were not mapped to the container) could confuse some (3d, media, compute) GPU driver version...

[tkatila]

While OCI spec specifies (non-mandatory) Linux specific nosymfollow option: https://github.com/opencontainers/runtime-spec/blob/main/config.md#linux-mount-options

And at least runc implements support for that: https://github.com/opencontainers/runc/blob/main/libcontainer/rootfs_linux.go#L462

Kubernetes device plugins API (Mount struct) does not include support for specifying that: https://github.com/kubernetes/kubelet/blob/master/pkg/apis/deviceplugin/v1beta1/api.proto#L190

So above 2 options are best that can be done, unless Kubernetes plugin API is improved.

Even though the DP API doesn't allow setting mount options, CDI's mount does. I tried to add the nosymfollow option to the mount request, but it didn't change anything. The symlink still resolved into the destination.

One option could be to use CDI's hooks to re-create the symlinks when the container is started. It would require more studying if it's possible.

While mapping whole by-path directory to container reveals PCI IDs and device file names of the other GPUs, I don't think that's any kind of security issues. But presence symlinks for the other GPUs (that were not mapped to the container) could confuse some (3d, media, compute) GPU driver version...

Yes. I ran some pytorch examples with the by-path directory fully mounted and they seemed to work fine. Though, it would require much more testing to be sure about every use case.

[tkatila]

I did some grepping on some of the key projects I found: Intel gpu runtime, oneccl, libfabric. In some projects the by-path items are directly opened and the code fails when the open fails => mounting the whole by-path directory is probably bad.