MSBuildWorkloadSdkResolver intermittently fails for custom MSBuildSdk package from local source

[matkoch]

Describe the bug

I'm testing my custom MSBuildSdk package. The procedure is roughly as follows:

• Several tests start running in parallel
• First is selected to package the custom MSBuildSdk package
• Once packaging is done, lock is released (timestamps verify this is working correctly)
• Tests are partially passing and failing
• Rerunning failing tests makes them successively pass

To Reproduce

Another contributor mentioned (2024), that there's a race condition: #42269 (comment)

Exceptions (if any)

error : Could not resolve SDK "Nuke.Build.Sdk". Exactly one of the probing messages below indicates why we could not resolve the SDK. Investigate and resolve that message to correctly specify the SDK.
error :   SDK resolver "Microsoft.DotNet.MSBuildWorkloadSdkResolver" returned null.
error :   Failed to resolve SDK 'Nuke.Build.Sdk/0.0.9999'. Package restore was successful but a package with the ID of "Nuke.Build.Sdk" was not installed.
The SDK 'Nuke.Build.Sdk/0.0.9999' specified could not be found.  /Users/matt/code/nuke3/lib/tests/build/Nuke.Build.Integration.Tests/bin/Debug/net10.0/_output/output/SdkTests/SingleFile/App.csproj

Further technical details

It happens way more often on CI than locally.

I tried a lot of things involving:

• Task.Delay
• DISABLEMSBUILDNODEREUSE and --disable-build-servers
• dotnet nuget locals http-cache/temp --clear
• --no-http and --no-http-cache etc.

Meanwhile I wrapper my code in some retry logic. It leads to much less failed CI runs, but that's not how I'd like to keep it:

async Task<Process> RunProcessWithRetry(int maxRetries = 3)
{
	var attempt = 0;
	while (true)
	{
		attempt++;
		var process = await RunProcess();

		// TODO(blocked): https://github.com/dotnet/sdk/issues/52765
		// SDK resolution can fail intermittently due to NuGet cache race conditions
		var shouldRetry = process.ExitCode != 0 && process.Output.ToText().Contains("was not installed") && attempt < maxRetries;
		if (!shouldRetry)
			return process;

		Output.WriteLine($"SDK resolution failed, retrying ({attempt}/{maxRetries})...");
		await Task.Delay(1000 * attempt);
	}
}