Track performance changes in CI

[adriencaccia]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Add a system to track performance over time, using the repository's existing benchmarks. This will help ensure that no unwanted performance regressions are merged, and give more visibility on performance improvement work like #6107

Describe the solution you'd like

Use https://codspeed.io/ with a new workflow that runs on every PR and the master branch.

Additional context

I already installed CodSpeed on a fork of arrow-rs, and you can see what it looks like on a PR that improves performance (based on #6145):
CodSpeedHQ#51
https://codspeed.io/CodSpeedHQ/arrow-rs/branches/fork-6145-samuelcolvin/improve-like-regex

Thanks @samuelcolvin for validating the need

[Xuanwo]

Hi, thanks for bringing this up! Many people are concerned about performance changes in arrow-rs.

I have a question about codspeed: Where do these performance tests take place? Are they run on GitHub Actions or on a dedicated environment? How can we ensure stable performance metrics and avoid false positives?

[adriencaccia]

I have a question about codspeed: Where do these performance tests take place? Are they run on GitHub Actions or on a dedicated environment?

They are run on GitHub Actions default runners, using https://github.com/CodSpeedHQ/action/.

How can we ensure stable performance metrics and avoid false positives?

Thanks to the action, the benchmarks are run on a simulated CPU and thus the measurements are stable (you can read more about it in our docs).

On the existing benchmarks, you can see that the results are quite consistent:

• On a sample PR https://codspeed.io/CodSpeedHQ/arrow-rs/branches/fork-6139-Rafferty97/simplify-struct-array
• On a specific benchmark (link to the page):
  

The links above is the CodSpeed integration of a forked arrow-rs.

[samuelcolvin]

Thanks @adriencaccia I'm strongly in favor of adding codspeed. I discussed this with @alamb and I think he was broadly in favor.

We've been using codspeed on pydantic-core since 2022, and it's absolutely key to maintaining and improving performance. See for example pydantic/pydantic-core#1155 (comment).

[Xuanwo]

Thanks to the action, the benchmarks are run on a simulated CPU and thus the measurements are stable (you can read more about it in our docs).

That's great, thanks a lot for the link.