Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] BE IO metrics are missing when storing data on nvme devices #48350

Open
2 of 3 tasks
xiacongling opened this issue Feb 26, 2025 · 0 comments
Open
2 of 3 tasks

[Bug] BE IO metrics are missing when storing data on nvme devices #48350

xiacongling opened this issue Feb 26, 2025 · 0 comments

Comments

@xiacongling
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

Version

2.1.7

What's Wrong?

When deploying Doris BE on machines with NVME disks, I/O metrics like doris_be_max_disk_io_util_percent or doris_be_disk_io_time_ms are all 0's.
After debugging, it is found that device labels in the metrics are like nvmeXnYp.

$ curl http://localhost:8040/metrics | grep "doris_be_disk_io_time_ms"
# TYPE doris_be_disk_io_time_ms counter
doris_be_disk_io_time_ms{device="nvme1n1p"} 0
doris_be_disk_io_time_ms{device="nvme5n1p"} 0
doris_be_disk_io_time_ms{device="nvme0n1p"} 0
doris_be_disk_io_time_ms{device="nvme3n1p"} 0
doris_be_disk_io_time_ms{device="nvme6n1p"} 0
doris_be_disk_io_time_ms{device="nvme7n1p"} 0
doris_be_disk_io_time_ms{device="nvme4n1p"} 0
doris_be_disk_io_time_ms{device="nvme2n1p"} 0

while the real device names are like nvmeXnY or nvmeXnYpZ.

$ cat /proc/partitions
major minor  #blocks  name

   8        0  468320256 sda
   8        1     488264 sda1
   8        2  467830784 sda2
 253        0   20971520 dm-0
 253        1   12582912 dm-1
 253        2  434274304 dm-2
 259        0 1875374424 nvme0n1
 259        1 1875373056 nvme0n1p1
 259        2 1875374424 nvme1n1
 259        3 1875373056 nvme1n1p1
 259        4 1875374424 nvme7n1
 259        5 1875373056 nvme7n1p1
 259        6 1875374424 nvme6n1
 259        7 1875373056 nvme6n1p1
 259        8 1875374424 nvme5n1
 259       12 1875373056 nvme5n1p1
 259        9 1875374424 nvme4n1
 259       13 1875373056 nvme4n1p1
 259       10 1875374424 nvme3n1
 259       14 1875373056 nvme3n1p1
 259       11 1875374424 nvme2n1
 259       15 1875373056 nvme2n1p1

What You Expected?

expose I/O metrics correctly for NVME devices

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant