inputs.nvidia_smi: many fields are parsed into the schema structs but never emitted

[81reap]

Use Case

I want to cutover my custom python exporter with Telegraf. I primarily use it to export host level metrics for my ML server. I like Telegraf as an option but there are a few limitations for Nvidia GPU monitoring.

1. Why did the GPU slow down, and for how long?

nvidia-smi reports <clocks_event_reasons_counters> and is the only way to attribute a throttle without catching it in the act. Reporting this helps catch issues that may happen between two log scrapes.

2. How close is the GPU to throttling?

On Ada and later, gpu_temp_tlimit reports the card's thermal margin in degrees. This changes based on the card and cooling policy and is more actionable than absolute temperature.

3. Is the bottleneck the bus?

rx_util / tx_util report host to device transfer saturation. This helps identify issues with a GPU sitting at low utilisation under an heavy load.

4. Misc metrics.

• average_power_draw -> instantaneous power_draw is noisy
• min / max / default / requested power limits -> detect a GPU running under a non-stock cap

Expected behavior

Fields that nvidia-smi reports and the plugin already parses are emitted on the
nvidia_smi measurement.

Actual behavior

1. Schema v13 parses ~25 fields it never emits

schema_v13/types.go fully defines <clocks_event_reasons_counters>, <temperature>, <gpu_power_readings> and the PCIe utilisation tags. schema_v13/parser.go writes only temperature_gpu and power_limit and nothing at all from the counters:

$ grep -c SetIfUsed plugins/inputs/nvidia_smi/schema_v13/parser.go
79
$ grep -c ClocksEvent plugins/inputs/nvidia_smi/schema_v13/parser.go
0

A fully populated block from nvidia-smi -q -x on this host:

<temperature>
    <gpu_temp>38 C</gpu_temp>
    <gpu_temp_tlimit>49 C</gpu_temp_tlimit>
    <gpu_temp_max_tlimit_threshold>-7 C</gpu_temp_max_tlimit_threshold>
    <gpu_temp_slow_tlimit_threshold>-2 C</gpu_temp_slow_tlimit_threshold>
    <gpu_temp_max_gpu_tlimit_threshold>0 C</gpu_temp_max_gpu_tlimit_threshold>
    <gpu_target_temperature>88 C</gpu_target_temperature>
    <memory_temp>N/A</memory_temp>
    <gpu_temp_max_mem_tlimit_threshold>N/A</gpu_temp_max_mem_tlimit_threshold>
</temperature>

Of which Telegraf emits exactly one field:

$ telegraf --test --config repro.conf | grep -o 'temperature[^,]*'
temperature_gpu=38i

2. Schema v12 has the same gap, for ~17 fields

schema_v12/types.go already declares GpuTempTlimit, SupportedGpuTargetTemp, RxUtil / TxUtil and the full GpuPowerReadings block, and schema_v12/parser.go writes none of them.

3. power_limit is missing entirely on some v12 cards

The v12 parser reads only GpuPowerReadings.PowerLimit, but many cards report <current_power_limit> inside <gpu_power_readings> and no <power_limit> tag. The v13 parser already handles both. These testdata are missing power_limit expectation in the test suite:

Fixture	power_limit
testdata/a100-sxm4-v12.xml	absent (card reports 500.00 W)
testdata/rtx-3060-v12.xml	absent (card reports 170.00 W)
testdata/rtx-3080-v12.xml	absent (card reports 336.00 W)

This is adjacent to #15279 but not the same case. That issue covered the RTX 3090, which uses the legacy <power_readings> block with a power_limit tag. Cards using <gpu_power_readings> with only current_power_limit were never wired up.

4. Absolute temperature thresholds are dropped at unmarshal on v13

The v13 Temperature struct declares only the *_tlimit_threshold tag. Cards reporting absolute thresholds use different tag names (gpu_temp_max_threshold, gpu_temp_slow_threshold, gpu_temp_max_gpu_threshold, gpu_temp_max_mem_threshold), which are discarded by the XML unmarshaler before any parser code runs.

The existing testdata/rtx-3080-v13.xml reports 98 C / 95 C / 93 C for those tags but none are collected.

5. Values above 2^31 are silently dropped on 32-bit builds

common.SetIfUsed parses integers with strconv.Atoi, which is documented as equivalent to ParseInt(s, 10, 0) — int-sized, so 32 bits on the 32-bit targets Telegraf ships (linux_i386, linux_armhf, freebsd_i386, windows_i386, per the Makefile). SetIfUsed assigns only when err == nil, so an out-of-range value produces no log line, no error, and no field:

$ GOARCH=386 GOOS=linux go run atoi.go   # strconv.Atoi("4251825415")
2147483647 strconv.Atoi: parsing "4251825415": value out of range

The throttle counters are microsecond accumulators, so they cross 2^31 after roughly 36 minutes of throttling. My current machine is already past it:

$ nvidia-smi -q -x | grep sw_power_cap
    <clocks_event_reasons_counters_sw_power_cap>4251825415 us</clocks_event_reasons_counters_sw_power_cap>

6. The README is outdated

Additional info

System: Telegraf 1.39.2, NVIDIA driver 595.84 (nvsmi_device_v13.dtd),
NVIDIA RTX 4000 SFF Ada Generation, Linux 6.18.40.

Config:

[[inputs.nvidia_smi]]

[81reap]

Raised some PRs

The STACK

I was hoping to try out the new github stacked PRs but it seems you can only do that for branches on the same repo, not a fork.

PR 1 :: #19405 :: fix(inputs.nvidia_smi): Parse integer fields as 64 bit
PR 2 :: #19406 :: feat(inputs.nvidia_smi): Add throttle counter, thermal, power and PCIe fields for schema v13
PR 3 :: #19407 :: feat(inputs.nvidia_smi): Add thermal, power and PCIe fields for schema v12
PR 4 :: #19408 :: docs(inputs.nvidia_smi): Document all emitted fields and measurements

[srebhan]

@81reap in general it is better to open one issue per item as this way we can directly link PRs to each issue in a 1:1 relation. This also allows to automatically close the issues when merging the PRs which makes our review life a lot easier as we don't need to manually close issues. Last but not least, individual issues are easier to discover for others having the same problem...

[81reap]

@srebhan just some context on how this ended up as stacked PRs.

Senior engineers I have interacted with IRL have all encouraged me to split my work into smaller commits I tend to fix and refactor small things that are semirelated or blockers to the core issue, which bloats the diff and makes things harder to review. Stacking let me break this +1,000 line change into 4 smaller PRs that I cross linked on GitHub.

I had hoped to use Github's new stacking feature so that to maintainers they would all appear in the UI as one PR. Unfortunatly, I was unable to raise it like this because I was unaware that they limit the feature to local branches only, not forks.

That said, if maintainers feel differently I am happy to adjust. There are more changes beyond this issue I wish to make to telegraf and I would rather make PRs that make maintainer's lives easier 😄 If its possible to give me read/write access to a feature branch called 81reap or something that I can try raising future changes as stacked PRs too!!

[81reap]

I just realized the PRs I have don't address the clock event counters. I will work on adding a 5th PR to the stack. following @srebhan 's advice I will address v12 and v13 schema changes all in one PR rather than splitting them up as I did with this.

[81reap]

I have raised draft pr #19452 as the final change needed to close out this issue.