Making a standalone build to run on multiple PC's for comparison

[Andrew900460]

I've read past issues that sounded similar to this topic, but it appeared that certain things were out of date or just didn't give the full picture.
I do understand that BenchmarkDotNet creates a separate program that is ran for each benchmark, so I get that there are some difficulties with making standalone builds due to that.

But it does sound like it is possible one way or another to achieve, otherwise, somone woud've plainly said "no this is currently is not possible".
To clarify, I was making a standard [Benchmark] test to time certain functions. And I was reading that depending on how old your CPU is, certain x86 instructions have gotten faster in newer architectures. My CPU is a Skylake, and I have a friend who's CPU is a few years newer than mine. So I was just curious how different the benchmark results would be on his CPU.

So I tried making a build in Visual Studio, and tried various different settings, and he would always get some crash message.
So then I read another older closed issues saying somthing about a "[InProcess........]" attribute, but it was a longer name.
That attribute did not show up through intellisence, and I don't think the documentation links that were given in other issues gave any sort of example of how to even use such an attribute either way. So I took it that the page was changed over time.
So I just tried adding [InProcess], which seemed to change somthing, but it still didn't work after sending my friend a new build.

I just wish that if this was indeed possible, if there was a clear example of how this is meant to be done. Like I said, I looked at old issues and the doc links, but the small bits that talked about "InProcess..." didn't really give a concrete example on how to properly do it.
It could be just somthing's not configured in the build settings, which would be my fault.

If this isn't at all possible, I'd try to make my own micro benchmark, but it seems I would need somthing more precise than StopWatch.

[timcassell]

Since you want to compare the exact same instructions, I assume you're building a NativeAOT application, and not the default CoreCLR. [InProcess] attribute uses the InProcessEmitToolchain which does not support NativeAOT (it requires JIT). You want to use the InProcessNoEmitToolchain. You can do this by passing a config to the benchmark runner.

DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance))

It is a less commonly used toolchain and slightly less accurate than the other toolchains, hence why there is no attribute for it. I'm not entirely sure why there are no docs for it, but I assume it's because we do not want to suggest a less accurate toolchain to users. It's for niche scenarios such as yours.

[Andrew900460]

Since you want to compare the exact same instructions, I assume you're building a NativeAOT application, and not the default CoreCLR. [InProcess] attribute uses the InProcessEmitToolchain which does not support NativeAOT (it requires JIT). You want to use the InProcessNoEmitToolchain. You can do this by passing a config to the benchmark runner.

DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance))
It is a less commonly used toolchain and slightly less accurate than the other toolchains, hence why there is no attribute for it. I'm not entirely sure why there are no docs for it, but I assume it's because we do not want to suggest a less accurate toolchain to users. It's for niche scenarios such as yours.

To be fair, I'm not quite sure if the generated exe is AOT or JIT.
I have been doing "Build -> Publish Selection"
And I have tried various settings like "Self-contained", "Produce single file", "Enable ReadyToRun compilation", "Trim unsued code", etc.

But I will try to add your example code, and see if that fixes it.

couple minutes later

Yea so I took your code, assigned the return value to a variable and passed it to BenchmarkRunner.Run
Currently seeing if I can get a build to run on a computer other than the one the code is on.

some more time later

Didn't seem to work on another computer.
If this is indeed how it's meant to look:

ManualConfig config = DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance));
BenchmarkRunner.Run<MyBenchmark>(config);

Then it could just be a matter of me screwing up the build settings and doing somthing that the package doesn't support, but I don't need either way.

[timcassell]

Didn't seem to work

That's not very descriptive. It would help if you could describe what actually happens. What does the log say?

[timcassell]

https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot

[Andrew900460]

Didn't seem to work

That's not very descriptive. It would help if you could describe what actually happens. What does the log say?

Right, sorry.

So from the build folder, we ran the exe in "publish/win-x64"
And the console showed this exception:

Unhandled exception. System.UnauthorizedAccessException: Access to the path 'C:\Windows\system32\BenchmarkDotNet.Artifacts\results' is denied.
at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor)
at System.IO.Directory.CreateDirectory(String path)
at BenchmarkDotNet.Extensions.CommonExtensions.CreateIfNotExists(String directoryPath)
.....

And so on. There was more to the error message, but I had to manually type that all out ^
I will say that the stack trace was executing my BenchmarkRunner.Run() function.
The rest of the stack trace was internal BenchmarkRunner functions and such.
Until it eventually got to CommonExtensions.CreateIfNotExists(String directoryPath)

I mean, it is confusing to me that it would be trying to create a folder in system32 even on my computer where it does work.

[Andrew900460]

https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot

I was also reading more about this, and not just that link (youtube videos about aot in C#)
So, I never added <PublishAot>true</PublishAot> to the project file. I only set the deployment mode to "self-contained", which apparently doesn't guarantee AOT, it just makes it so that all your dependencies are in the build folder.

[timcassell]

I believe the BenchmarkDotNet.Artifacts folder is created at the working directory. It would seem that system32 is the working directory there.

[Andrew900460]

I believe the BenchmarkDotNet.Artifacts folder is created at the working directory. It would seem that system32 is the working directory there.

Right. Well I'm fairly certain my friend (who is helping me test all this) didn't run the exe inside their system32.
It's unfortunate that this trouble shooting is a bit clunky.

[timcassell]

I don't know how that would happen, but you should be able to override the path with [ArtifactsPath("path")]. Or config.WithArtifactsPath("path").

[Andrew900460]

I think we may have figured out what the issue was. But I still have to hear back from my friend so I can figure out what it was.
I will update this comment when I hear back.

[Andrew900460]

I don't know how that would happen, but you should be able to override the path with [ArtifactsPath("path")]. Or config.WithArtifactsPath("path").

We seemed to have fixed the issue.
I did not need to use that config.WithArtifactsPath("path") bit you mentioned.

I think it was a combination of many pitfalls:

1. Me not understanding the export path when publishing the build (I was getting it confused with the "bin/Release/net5.0" executable. So I moved the export path somwhere else so I knew I was sending the right thing).
2. It seems the "Deployment mode" setting does need to be set to "self-contained" for this to work (out of the box).
3. My friend had an anti-virus that would freak out when running the exe (understandably). And we suspected that it was taking the exe and putting it in the system32 folder before running it (why? no idea! that seems strange)

Other final conclusions for future readers
Not sure if DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance))
Had any influence on fixing the problem, I assume so though.

Also, I never ended up using <PublishAot>true</PublishAot>, to be clear. Though, it would probably be good for certain types of benchmarks.

[timcassell]

Great, glad you got it figured out!

Also, I never ended up using <PublishAot>true</PublishAot>, to be clear. Though, it would probably be good for certain types of benchmarks.

If you're not using that, it's likely CoreCLR is used, which JIT compiles assembly for the current machine, thus you are possibly not comparing exact instructions.