Improving a Competitor's Numbers (On Purpose)

[giscus[bot]]

Improving a Competitor's Numbers (On Purpose)

Quarkus: Supersonic Subatomic Java

https://quarkus.io/blog/fairness-in-benchmarking/

[vsevel]

I can setup a call once I am back from vacation for that one (25 Aug)

@franz1981 I will be away when you are back. this can wait mid september. thanks for the proposition.

a bit of context in the meantime. we are starting to look at auto scaling. I am interested in figures comparing the different technologies we currently allow, namely quarkus, springboot and nodejs. since you provide already a nice framework for quarkus and springboot, I was wondering how much effort that would be to support nodejs.

I remember in the early days, you had this slide with a density picture with jvm, node.js and go:
https://youtu.be/SQDR34KoC-8?t=1610

we are starting to have the discussion internally about whether or not node.js is a good fit for auto-scaling. there is much to consider (e.g. functional coverage), but performance and density are 2 important requirements. I am wondering how easy that would be to fit node.js into this benchmark.

[franz1981]

@holly-cummins and @edeandrea too

Let's say that we can still have conversations about it, but my experience as an engineer is that...it depends.
NodeJS without an ORM which is comparable to what Quarkus or Spring could offer, could be a rather different beast and clearly not Apple to apple to compare against, despite interesting.

[holly-cummins]

We've had similar discussions about .NET. It would be really cool to do all of these benchmarks, and there's definitely demand for the numbers. But we're cautious about expanding past the core comparison that we started with, because of maintenance concerns – even maintaining the two/three versions of the app we have is a bit arduous. As soon as one version lags behind the others and misses a code change, the benchmark for that platform becomes useless. Lab time for execution is also a problem, since the more variants we have, the longer each one takes.

[vsevel]

I understand. I was not asking for you to do it, although if you did I would not mind ;-)
I got confused between the 2 articles from Francesco . I though there was another (even simpler) benchmark I could use as a base. but that was based on netty, not quarkus.
and so I am back to the original benchmark. and like it was said by Francesco, there are some questions about comparing quarkus/SB with a full blown ORM, and node.js, which does not have.
I am going to try coming up with the node app. I will keep you posted,

note: I experienced 2 strange behaviors when trying to replicate the benchmark.
the first one is about configuring and launching.
I was expecting to be able to set env variables (e.g. ITERATIONS, ...) and launch ./scripts/perf-lab/run-benchmarks.sh at the root.
if I do that:

export ITERATIONS=1
./scripts/perf-lab/run-benchmarks.sh

I get:

#####################
Configuration Values:
  CPUS_APP=0-3
  CPUS_DB=4-6
  CPUS_OTEL=7-9
  CPUS_LOAD_GEN=10-12
  CPUS_MONITORING=13
  CPUS_FIRST_REQUEST=10
  GRAALVM_HOME:
  GRAALVM_VERSION: 25.0.2-graalce
  HOST: LOCAL
  ITERATIONS: 3
...

Using installed jbang (0.139.3)
18:08:47.741 Running qDup version 0.11.2
18:08:47.937 Error: cannot find ./main.yml
18:08:47.938 Error: cannot find ./helpers/

I was expecting env variables to be used.
and I was expecting to be able to run from the base directory.
am I missing something?

the second point was that my benchmark would die at some point.
claude suggested the following change in main.yaml:
instead of
${{LOAD_TESTS_SCRIPTS_DIR}}/load-test-fixed-threads-read-all.hf.yml &>${{REPO_DIR}}/logs/hf-${{RUNTIME_NAME}}-${{ITERATION}}.log &
do:

                ${{LOAD_TESTS_SCRIPTS_DIR}}/load-test-fixed-threads-read-all.hf.yml \
                </dev/null &>${{REPO_DIR}}/logs/hf-${{RUNTIME_NAME}}-${{ITERATION}}.log &

with the following explanation:

We added </dev/null because the Hyperfoil load-test launch was silently hanging (not crashing) on every run, and it was causing spurious ABORTED=true aborts.

The bug

That line launches run@hyperfoil as a background job (&) with stdout/stderr redirected to a log file — but stdin was left attached to the terminal. qDup then does, across separate sh: steps:
...

if that makes sense I can create an issue.

[vsevel]

in the context of auto-scaling I am interested in density. If I look at the published results https://github.com/quarkusio/spring-quarkus-perf-comparison/#published-results I see:

• quarkus native: 5437 tps / 98 Mb = 55 tps/Mb
• quarkus virtual: 15299 / 292 Mb = 52 tps/Mb
  That seems very close, yet the JVM will spend some cpu cycles at every start on C1/C2, whereas quarkus native might be less optimized but will not have to go through this. in the end I would expect quarkus native to be more cost efficient, specifically on the auto scaling use case where we will have many many restarts.
  I do not know if this a correct assumption. and if so, I am wondering if that can be measured, and reflected in the benchmark.

[holly-cummins]

I've got some plots cooking for density, for both high-load and low-load scenarios. One is merged but not publicised, and the other still needs to be merged. (My starting assumption is that in low load scenarios memory is what determines maximum density, and in high-load scenarios throughput determines density. Obviously, there's a messy middle where we can't say in advance which would be the limiting factor, and it might even be different for different frameworks, but I haven't gone to that level of complexity.)

This is the plot for the high-load case. It ends up looking a bit like the throughput chart, in practice:quarkusio/benchmarks#484

The memory density chart is a bin packing chart, so visually more exciting: https://quarkus.io/benchmarks/tuned/spring-quarkus-perf-comparison-latest-tuned-density-by-rss-for-main-comparison-light.png

[vsevel]

to illustrate what I was saying, I added different runtimes I combined some of the metrics with some weight to get a score.
in there quarkus native (score 21) appears slightly below springboot 4 virtual (26), which does not take into account that sb jvm needs heavy warmup, which graalvm does not. I am not sure how to reflect this.
do not look at the absolute numbers. I got them on my 6 cores dev machine. still waiting for dedicated host to replicate the benchmark.
yet you see the idea. if this is a long running process, the warm up cost can be diluted into the process's activity over a long period of time.
but if we are talking about auto-scaling with pods that come and go very often, then the warm up phase takes a bigger proportion on the process efficiency.

Method: each metric normalized to 100 relative to the best performer in this reduced set (memory/TTFR: best/value×100; throughput: value/best×100), then 0.35×Throughput + 0.35×MemoryUnderLoad + 0.20×TTFR + 0.10×StartupMemory.

┌──────┬──────────────────┬───────────┬──────────────────────┬─────────────────────────┬──────────────────┬───────────────────┬───────┐
│ Rank │     Runtime      │ TTFR (ms) │ Startup memory (MiB) │ Memory under load (MiB) │ Throughput (tps) │ Density (tps/MiB) │ Score │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 1    │ rust-orm         │ 33.6      │ 7.15                 │ 12.16                   │ 3890.1           │ 319.97            │ 85.31 │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 2    │ go-orm           │ 58.5      │ 19.28                │ 38.20                   │ 3740.2           │ 97.91             │ 45.88 │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 3    │ quarkus3-virtual │ 3431.6    │ 243.44               │ 463.93                  │ 6702.5           │ 14.45             │ 36.41 │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 4    │ spring4-virtual  │ 9845.5    │ 440.85               │ 564.53                  │ 4817.2           │ 8.53              │ 26.14 │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 5    │ quarkus3-native  │ 276.6     │ 92.57                │ 246.71                  │ 3121.9           │ 12.65             │ 21.23 │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 6    │ spring4-native   │ 1386.1    │ 326.23               │ 391.58                  │ 1718.1           │ 4.39              │ 10.76 │
├──────┼──────────────────┼───────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────────┼───────┤
│ 7    │ nodejs-orm       │ 2061.2    │ 147.30               │ 231.63                  │ 693.2            │ 2.99              │ 6.27  │
└──────┴──────────────────┴───────────┴──────────────────────┴─────────────────────────┴──────────────────┴───────────────────┴───────┘

[vsevel]

I am running on a WSL Ubuntu distro running on Windows 11, which is itself running as a VM, running in a private datacenter. I know this is not as solid as a physical dedicated perf machine, but that is what I have available at this point, and even if the numbers could be more accurate, hopefully there is still some value.
those early results were obtained with 6 cores. I have now bumped to 12 cores. I allocate 2 cores to the app, and rest according to recommendation.

the view of the WSL is:

$ lscpu | grep -iE "thread|core|socket|cpu\(s\)" | head -8
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Thread(s) per core:                      1
Core(s) per socket:                      12
Socket(s):                               1
NUMA node0 CPU(s):                       0-11

$ lscpu | grep -i numa; echo "--- distribution of CPU per node ---"; ls -d /sys/devices/system/node/node* 2>/dev/null | sed 's|.*/||' | while read n; do echo "  $n: $(cat /sys/devices/system/node/$n/cpulist 2>/dev/null)"; done
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
--- distribution of CPU per node ---
  node0: 0-11

I do not have information on the underlying hardware at this point.

[franz1981]

No worries, just beware that the warmup period change dramatically with just 2 cores to the app as this article shows. The JIT compiler need time and some idleness to make sure both frameworks are fully compiled

[vsevel]

so should the warmup period be extended when running on 2 cores? even further with 1 core (which is the config I am running right now)?
FYI I am looking at getting some numbers for our auto scaling use case, not for our static servers. that is why I decided to focus on lower cpu numbers.
my primary target is density tps/Mb. For memory I decided to use the RSS under load, and not the RSS after startup, or after 1st request. it tends to lower the difference between quarkus and springboot.
I did several tests also with a lower Xmx, and no Xms. I realized that a large Xmx, plus some heavy load would favor high throughput but consume more memory, hence lowering the density. I got the best results with 128M, even for SB suprisingly.

[franz1981]

so should the warmup period be extended when running on 2 cores? even further with 1 core (which is the config I am running right now)?

I have a pending PR which perform multiple warmups rounds because just hammering the app for longer sometime won't help; the pr is referenced in this article.

it tends to lower the difference between quarkus and springboot.

The tps are still different, and without using Shenandoah compact, Semeru/OpenJ9 compact, or any aggressive trimming, a JVM tends to use as much as given, not to stay compact - which is expected.
Under load GC just use as much as needed, and RSS risk to be a sterile metrics, despite I know is commonly used on containers, but my 2c is that you need a proper GC settings to stay compact, and regardless need to be normalized against the peak tps.

I got the best results with 128M, even for SB suprisingly.

That's surprising but I won't make any guess without JFR data, see quarkusio/spring-quarkus-perf-comparison#521

[vsevel]

I have created a discussion here: quarkusio/spring-quarkus-perf-comparison#712

[vsevel]

what memory field do you report? rss.startup, rss.firstRequest or load.rss?
I am under the impression that you use rss.startup or rss.firstRequest. is that correct?
if true, I think that would be more fair to use load.rss. typically in my numbers above, we have rss.startup at 243 Mb for quarkus jvm and 440 for SB 4 jvm (so 200 Mb gap), whereas under load the gap shortens to only 10 Mb (463 Mb in quarkus jvm and 560 on SB 4 jvm).
so quarkus numbers are better in absolute terms, but it is not as bad as it looks when you look under load.

when you calculate the TTFR, you do not apply warm up, right ? you start the process and send one request? in the measurement do you include launching the process plus getting the first response?