feat: add support for analysis mode #152

not-matthias · 2025-11-19T18:32:08Z

No description provided.

codspeed-hq · 2025-11-20T17:57:58Z

CodSpeed Performance Report

Merging #152 will degrade performances by 10%

_{Comparing cod-1711-codspeed-rust-add-analysis-mode-support (435a805) with main (9181bce)}

Summary

⚡ 31 improvements
❌ 2 regressions
✅ 333 untouched
🆕 2 new

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Mode	Benchmark	`BASE`	`HEAD`	Change
⚡	WallTime	`large_drop`	204 µs	170.9 µs	+19.35%
⚡	WallTime	`large_setup`	20 ns	11 ns	+81.82%
⚡	WallTime	`small_setup`	2 ns	1 ns	×2
❌	WallTime	`iter_with_setup`	36 ns	39 ns	-7.69%
❌	WallTime	`iter_batched_per_iteration`	36 ns	40 ns	-10%
⚡	WallTime	`init_array[1000]`	2.1 µs	2 µs	+7.34%
🆕	Simulation	`allocate`	N/A	< 1 ns	N/A
🆕	WallTime	`allocate`	N/A	1 s	N/A
⚡	WallTime	`slice_into_vec_with_bytes`	59 ns	55 ns	+7.27%
⚡	WallTime	`string_copy_with_bytes_counter`	61 ns	58 ns	+5.17%
⚡	WallTime	`print_env_hello`	5.2 µs	2 µs	×2.6
⚡	WallTime	`recursive[5]`	34 ns	33 ns	+3.03%
⚡	WallTime	`recursive_memoized[BTreeMap<u64, u64>, 20]`	2.3 µs	1.8 µs	+23.7%
⚡	WallTime	`recursive_memoized[BTreeMap<u64, u64>, 30]`	3.4 µs	2.9 µs	+15.11%
⚡	WallTime	`recursive_memoized[HashMap<u64, u64>, 0]`	16 ns	15 ns	+6.67%
⚡	WallTime	`recursive_memoized[HashMap<u64, u64>, 10]`	1.8 µs	1.5 µs	+16.29%
⚡	WallTime	`recursive_memoized[HashMap<u64, u64>, 20]`	3.8 µs	3.3 µs	+15.58%
⚡	WallTime	`recursive_memoized[HashMap<u64, u64>, 30]`	6.5 µs	5.7 µs	+13.69%
⚡	WallTime	`generate_combinations[5]`	1,181 ns	794 ns	+48.74%
⚡	WallTime	`generate_combinations[6]`	1.8 µs	1.4 µs	+30.46%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Copilot

Pull request overview

This PR adds support for a new "analysis" measurement mode to the CodSpeed benchmarking framework. The analysis mode enables instrumentation similar to simulation mode and integrates with instrument hooks for tracking benchmark execution.

Key changes:

Introduced a new Analysis variant to the MeasurementMode enum
Integrated InstrumentHooks for tracking benchmark lifecycle events
Removed the deprecated is_instrumented() function and RunningOnValgrind enum variant

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
crates/divan_compat/examples/benches/alloc.rs	New benchmark file for testing memory allocation scenarios
crates/divan_compat/examples/Cargo.toml	Added the new alloc benchmark to the build configuration
crates/codspeed/src/request/mod.rs	Removed unused `RunningOnValgrind` client request variant
crates/codspeed/src/measurement.rs	Removed deprecated `is_instrumented()` function
crates/codspeed/src/codspeed.rs	Integrated `InstrumentHooks` for benchmark lifecycle tracking
crates/cargo-codspeed/src/measurement_mode.rs	Added `Analysis` variant to measurement modes
crates/cargo-codspeed/src/build.rs	Extended codspeed cfg flag to apply to analysis mode
.github/workflows/ci.yml	Added CI workflow for testing memory analysis integration
crates/codspeed/instrument-hooks	Updated submodule commit reference

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/divan_compat/examples/benches/alloc.rs

crates/codspeed/src/codspeed.rs

GuillaumeLagrange

olgtm

GuillaumeLagrange · 2025-11-25T03:03:43Z

.github/workflows/ci.yml

+      #     runner-branch: cod-1670-runner-profile-memory-of-command
+      #     run: cargo codspeed run
+      #     mode: memory
+      #     token: ${{ secrets.CODSPEED_TOKEN }}


We should be able to remove secrets now with OIDC

GuillaumeLagrange · 2025-11-25T03:06:56Z

.github/workflows/ci.yml

          token: ${{ secrets.CODSPEED_TOKEN }}

+
+  compat-integration-test-memory:


We can have multi-dimensional matrices, as well as some dimensions precising two things

We could combine everything in one big matrix, one dimension being the library we are testing, the other one the mode and the runner it runs on like here https://github.com/CodSpeedHQ/codspeed-cpp/blob/f23b68b27f8543f88f95ff0bdc46785733dee061/.github/workflows/ci.yml#L57-L62

WDYT ?

We could conditionnallly remove the .cargo/config.toml, as well as conditionnally skip the uploading part for memory (for now)

GuillaumeLagrange · 2025-11-25T03:08:24Z

crates/codspeed/src/codspeed.rs

 impl CodSpeed {
    pub fn new() -> Self {
-        let is_instrumented = measurement::is_instrumented();
+        let ih = InstrumentHooks::instance();


not a fan of abbreviations overall, but here it does not bring that much, how about hooks ? or hooks_instance or instrument_hooks_instance ?

GuillaumeLagrange · 2025-11-25T03:10:00Z

crates/codspeed/src/codspeed.rs

    pub fn start_benchmark(&mut self, name: &str) {
        self.current_benchmark = CString::new(name).expect("CString::new failed");
+        let _ = InstrumentHooks::instance().start_benchmark();
        measurement::start();


will that not confuse valgrind to keep both this and the start_benchmark call when built in simulation ?

It might be the time to say goodbye to the inline assembly and make a major out of this

GuillaumeLagrange · 2025-11-25T03:10:07Z

crates/codspeed/src/codspeed.rs


    #[inline(always)]
    pub fn end_benchmark(&mut self) {
        measurement::stop(&self.current_benchmark);


same remark here

GuillaumeLagrange · 2025-11-25T03:13:17Z

crates/cargo-codspeed/src/build.rs

            "-Cstrip=none".to_owned(),
        ];

        // Add the codspeed cfg flag if simulation mode is enabled


Update comment, or create a function that takes measurement_mode and outputs a bool, named should_build_benchmark_target_to_run_only_once or something like this, to convey the intention

not-matthias changed the title ~~chore: wip [skip ci]~~ feat: add support for analysis mode Nov 20, 2025

not-matthias force-pushed the cod-1711-codspeed-rust-add-analysis-mode-support branch from 8638bef to 5bab4b1 Compare November 21, 2025 15:26

not-matthias added 2 commits November 21, 2025 16:31

feat: add memory profiling in CI

0ce0cab

feat: add support for analysis mode

5a5d450

not-matthias force-pushed the cod-1711-codspeed-rust-add-analysis-mode-support branch from 5bab4b1 to f2e74d7 Compare November 21, 2025 15:33

not-matthias requested a review from Copilot November 21, 2025 15:33

Copilot AI reviewed Nov 21, 2025

View reviewed changes

crates/divan_compat/examples/benches/alloc.rs Outdated Show resolved Hide resolved

crates/divan_compat/examples/benches/alloc.rs Outdated Show resolved Hide resolved

crates/codspeed/src/codspeed.rs Show resolved Hide resolved

crates/codspeed/src/codspeed.rs Show resolved Hide resolved

not-matthias marked this pull request as ready for review November 24, 2025 09:52

not-matthias added 3 commits November 24, 2025 11:26

chore: add example with allocations

97e21d0

feat(cargo-codspeed): add analysis build mode

8d90e07

chore: bump instrument-hooks

435a805

not-matthias force-pushed the cod-1711-codspeed-rust-add-analysis-mode-support branch from f2e74d7 to 435a805 Compare November 24, 2025 10:26

not-matthias requested a review from GuillaumeLagrange November 24, 2025 10:27

GuillaumeLagrange requested changes Nov 25, 2025

View reviewed changes

		token: ${{ secrets.CODSPEED_TOKEN }}


		compat-integration-test-memory:

feat: add support for analysis mode #152

Are you sure you want to change the base?

feat: add support for analysis mode #152

Uh oh!

Conversation

not-matthias commented Nov 19, 2025

Uh oh!

codspeed-hq bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #152 will degrade performances by 10%

Summary

Benchmarks breakdown

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codspeed-hq bot commented Nov 20, 2025 •

edited

Loading