Using an accurate benchmark tool

[waghanza]

We are currently using wrk. This tool is great, no doubt about that. However, the purpose of such a tool is to check how a framework could perform at its best (grab the maximum performance it could give).

This project aims to reflect how frameworks could perform in real world scenarios (for example, using constant throughput). For this reason, I'm considering switching this tool.

Candidates are :

• https://github.com/tsenart/vegeta
• https://github.com/tag1consulting/goose
• https://github.com/codesenberg/bombardier
• https://github.com/sharkdp/hyperfine

Goose seems accurate for me. Any suggestions @dominikzogg @jeremyevans @panesofglass @jodosha @timriley @ahopkins @appleboy @yusukebe @p8 @mcollina @fundon @ohler55 @sy-records @enghitalo @lukaszlenart ?

PS : The idea is to change this tool on Q2 2025

PSS : Sorry to ping some individuals, I do not know how to ping the whole team at once 😇

[waghanza]

Sorry : not hyperfine (more to benchmark cli tools)

[fundon]

https://github.com/hatoo/oha
It is also a good testing tool.

[waghanza]

Seems more like a PoC for me hatoo/oha#617

But to be honest, it's promising

[appleboy]

Yes, tsenart/vegeta is a versatile HTTP load testing tool that can handle real-world scenarios, including constant throughput, which is what you're looking for. Vegeta is designed to be both a command-line utility and a library, making it flexible for various use cases.

Key Features of Vegeta

1. Constant Throughput: Vegeta can maintain a constant request rate, which is useful for simulating real-world traffic patterns.
2. Ease of Use: It has a simple command-line interface and can be easily integrated into scripts and CI/CD pipelines.
3. Detailed Reporting: Vegeta provides detailed metrics and can generate reports in various formats, including JSON and HTML.
4. Flexibility: It supports multiple attack modes, including rate-limited and burst modes, and can handle complex scenarios with custom headers, bodies, and methods.

[waghanza]

Also, how can vegeta handle constant throughput ? I mean, how can we make an equal sign between constant throughput and a real world scenario, there is no control about throughout

[waghanza]

Vegeta use 50rps per default, how can we define the rate to use facing the scenario we use ?

[waghanza]

I have tested vegeta (without rate limit), just to compare to wrk (since it has not rate limit) , and I can not explain result I've found in tsenart/vegeta#720

[waghanza]

How can I use constant throughput in vegeta ?

How can I use rate param in vegeta ? Nad this it really accurate in benchmark context @appleboy

cc @akyoto

[akyoto]

How can I use constant throughput [...] Nad this it really accurate in benchmark context

Constant throughput is useful in private benchmarks at a company with very clear constraints.
Constant throughput should be avoided for public benchmarks.

[mcollina]

I think wrk is ok. wrk2 is better ;).

[waghanza]

I do not understand how using constant throughput could be helpful.

I mean, with constant throughput approach, we must know the rate we should use. With wrk2, the rate used (by default) is 1000, but how can we know that 1000 is accurate for our scenario ? Or define any rate for it ?

[waghanza]

wrk2 is not yet compilable for arm

[yusukebe]

I prefer to bombardier, though I don't have any concrete reason. My favorite benchmark by @SaltyAom is using it:

https://github.com/SaltyAom/bun-http-framework-benchmark

[waghanza]

I there a way to enable any rate limit with it @yusukebe @SaltyAom ?

This feature is important, imho, to make more realistic results

[alanmsant2]

I use rewrk (Rust) link

[waghanza]

Any rationales behind ?

[alanmsant2]

In my perception, I found the results more accurate with Rewrk

[panesofglass]

I've only used bombardier, and I thought it did a good job. I'll have to try the others, but I cannot offer any good comparisons.