EPYC cpu running on lower frequencies than requested by a cpufreq governor when one of cpus inside CCX is on max frequency

[martin-mat]

Environment:
CPU:
Family: 0x19
Model: 0x11
Model name: AMD EPYC 9654 96-Core Processor
Stepping: 1
Microcode: 0xa101148

driver: amd-pstate passive
Highest performance: 3.74 GHz
Nominal performance: 2.4 GHz
Lowest non-linear frequency: 1.9 GHz
Lowest frequency: 400 MHz

Ubuntu 24.04 LTS
kernel 6.8.0.-51-generic

Observed behavior:
Real CPU frequencies of cores inside CCX (8 cores in case of this cpu) are lower that those requested by a cpufreq governor in case that one of cores inside this CCX is on a maximal turbo frequency.
Observed frequencies (MHz):

| Requested frequency | Real frequency |
-------------------------------------------------------------------------------------------
|                1600 |           1477 |
|                1800 |           1641 |
|                2100 |           1847 |
|                2400 |           2111 |
|                2900 |           2462 |
|                3700 |           2955 |
|                3709 |           3694 |

That looks like in this situation ( max turbo frequency on one of cores), P steps are very rough and especially in boost area it is either 2.9 GHz or 3.7 Ghz, no steps in between. In case it is really so for some reason, I think it would be better to set rather higher frequencies than lower compared to requested frequencies.

How to reproduce and observe this behavior:

• set all cpus inside a CCX (let's take logical cpus 8-15 and their siblings 200-207) to powersave governor with minimal frequency 400 MHz

# cpupower -c 8-15,200-207 frequency-set -g powersave

• check busy frequencies of cpus (Bzy_MHz)

# turbostat -c 8-15,200-207

they should be all on (or close to) 400 MHz (and they are).

• set frequency of one of the cpu inside the CCX to the maximal boost frequency, 3709 MHz in case of this CPU. For example by keeping powersave governor and increasing minimum frequency to the max boost frequency:

# cpupower -c 8 frequency-set --min 3709000

• set frequency of another cpu inside the CCX to a boost frequency lower than max boost, for example 3700 MHz. For example by keeping powersave governor and increasing minimum frequency:

# cpupower -c 9 frequency-set --min 3700000

• check busy frequencies by turbostrat

# turbostat -c 8-15,200-207

Expected behavior after this:

• Bzy_MHz for cpus 8,200 should be around 3709 MHz (and it is - ok)

• Bzy_MHz for cpus 9,201 should be around 3700 MHz (and it is not - in this case it is around 2955 MHz)

• set frequency of cpu 9 to boost max

# cpupower -c 9 frequency-set --min 3709000

• check busy frequencies by turbostat

# turbostat -c 8-15,200-207

Expected behavior after this:

• Bzy_MHz for cpus 8,200 should be around 3709 MHz (and it is - ok)
• Bzy_MHz for cpus 9,201 should be around 3709 MHz (and it is - ok in this case!)

Note that this behavior is not specific for a cpufreq governor, it is also obvervable when using userspace or other governors.