dotnet workload restore failing in ADO pipeline

[AlexandruRus23]

Describe the bug

Running dotnet workload restore in latest Azure Devops Windows 2022 agent image fails with error

Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Could not find workload 'microsoft-net-runtime-android' extended by workload 'android' in manifest 'microsoft.net.sdk.android' [C:\Program Files\dotnet\sdk-manifests\6.0.300\microsoft.net.sdk.android\WorkloadManifest.json]

To Reproduce

While targeting a Windows 2022 build agent (works fine on ubuntu) use a dotnet cli task to do a restore workloads step for an solution that contains a .Net 8 Web application and an Aspire Host.

• task: DotNetCoreCLI@2
  displayName: "Restore Workload"
  inputs:
  command: "custom"
  custom: "workload"
  arguments: "restore -s https://api.nuget.org/v3/index.json"

Exceptions (if any)

C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018: Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Could not find workload 'microsoft-net-runtime-android' extended by workload 'android' in manifest 'microsoft.net.sdk.android' [C:\Program Files\dotnet\sdk-manifests\6.0.300\microsoft.net.sdk.android\WorkloadManifest.json]
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.<>c__DisplayClass29_0.<<EnumerateWorkloadWithExtends>g__EnumerateWorkloadWithExtendsRec|0>d.MoveNext()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at System.Linq.Enumerable.AppendPrependIterator`1.LoadFromEnumerator()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at System.Linq.Enumerable.AppendPrepend1Iterator`1.MoveNext()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at System.Linq.Enumerable.All[TSource](IEnumerable`1 source, Func`2 predicate)
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetAvailableWorkloadDefinitions()+MoveNext()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at System.Linq.Enumerable.SelectEnumerableIterator`2.ToList()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetWorkloadSuggestionForMissingPacks(IList`1 packIds, ISet`1& unsatisfiablePacks)
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.NET.Build.Tasks.ShowMissingWorkloads.ExecuteCore()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Execute()
C:\Program Files\dotnet\sdk\8.0.404\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets(23,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(TaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)

Further technical details

This only happens on windows 2022 agents and it works on ubuntu

Output of dotnet --info

Windows 2022:

.NET SDK:
Version: 8.0.404
Commit: 7b190310f2
Workload version: 8.0.400-manifests.95994a6b
MSBuild version: 17.11.9+a69bbaaf5

Runtime Environment:
OS Name: Windows
OS Version: 10.0.20348
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\8.0.404\

.NET workloads installed:

Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Could not find workload 'microsoft-net-runtime-android' extended by workload 'android' in manifest 'microsoft.net.sdk.android' [C:\Program Files\dotnet\sdk-manifests\6.0.300\microsoft.net.sdk.android\WorkloadManifest.json]
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.<>c__DisplayClass29_0.<<EnumerateWorkloadWithExtends>g__EnumerateWorkloadWithExtendsRec|0>d.MoveNext()
   at System.Linq.Enumerable.AppendPrependIterator`1.LoadFromEnumerator()
   at System.Linq.Enumerable.AppendPrepend1Iterator`1.MoveNext()
   at System.Linq.Enumerable.All[TSource](IEnumerable`1 source, Func`2 predicate)
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetAvailableWorkloadDefinitions()+MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.Contains[TSource](IEnumerable`1 source, TSource value, IEqualityComparer`1 comparer)
   at Microsoft.DotNet.Workloads.Workload.VisualStudioWorkloads.GetInstalledWorkloads(IWorkloadResolver workloadResolver, InstalledWorkloadsCollection installedWorkloads, Nullable`1 sdkFeatureBand)
   at Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.AddInstalledVsWorkloads(IEnumerable`1 sdkWorkloadIds)
   at Microsoft.DotNet.Cli.WorkloadCommandParser.ShowWorkloadsInfo(ParseResult parseResult, WorkloadInfoHelper workloadInfoHelper, IReporter reporter, String dotnetDir, Boolean showVersion)
   at Microsoft.DotNet.Cli.CommandLineInfo.PrintWorkloadsInfo()
   at Microsoft.DotNet.Cli.CommandLineInfo.PrintInfo()
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

Host:
Version: 9.0.0
Architecture: x64
Commit: 9d5a6a9aa4
RID: win-x64

.NET SDKs installed:
6.0.136 [C:\Program Files\dotnet\sdk]
6.0.203 [C:\Program Files\dotnet\sdk]
6.0.321 [C:\Program Files\dotnet\sdk]
6.0.428 [C:\Program Files\dotnet\sdk]
8.0.111 [C:\Program Files\dotnet\sdk]
8.0.206 [C:\Program Files\dotnet\sdk]
8.0.307 [C:\Program Files\dotnet\sdk]
8.0.404 [C:\Program Files\dotnet\sdk]
9.0.101 [C:\Program Files\dotnet\sdk]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
X:\agent\8\s\global.json

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

Ubuntu 22.04

.NET SDK:
Version: 8.0.404
Commit: 7b190310f2
Workload version: 8.0.400-manifests.b9a0937d
MSBuild version: 17.11.9+a69bbaaf5

Runtime Environment:
OS Name: ubuntu
OS Version: 22.04
OS Platform: Linux
RID: linux-x64
Base Path: /usr/share/dotnet/sdk/8.0.404/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
[aspire]
Installation Source: SDK 8.0.400
Manifest Version: 8.2.2/8.0.100
Manifest Path: /usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.aspire/8.2.2/WorkloadManifest.json
Install Type: FileBased

Host:
Version: 8.0.11
Architecture: x64
Commit: 9cb3b725e3

.NET SDKs installed:
6.0.428 [/usr/share/dotnet/sdk]
8.0.404 [/usr/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.11 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.36 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.11 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
None

Environment variables:
Not set

global.json file:
/datadrive/agent/13/s/global.json

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

Note

Documentation mentions dotnet workload restore is looking at a solution and figuring out what it needs to restore. That doesn't seem to happen as we're only doing the restore for the aspire workload.

I only noticed after collecting all of this information that not even dotnet --info works in the agent now

[marcpopMSFT]

Something is wrong with your windows install. Any chance you could try repairing the 8.0.404 install? From the error, you're using hte 8.0.404 SDK but finding a 6.0.300 workload which shouldn't be possible. That would indicate that C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.android is missing for some reason. I'm hoping that a repair will help.

If that doesn't work, you can try installing aspire directly and using a version of the workloads to install (dotnet workload install aspire --version 8.0.404)

[bdach]

For reference I have started seeing similar failures on github actions runners, and I would not be surprised in the slightest if the ADO failures were caused by the same sort of fault with the ADO pipeline runner image because the symptom I'm seeing on github actions is exactly the same.

Reported elsewhere as actions/runner-images#11402.

[bdach]

Well for better or for worse the investigation into the runner images appears to have hit a dead end and I have been told to go back here (actions/runner-images#11402 (comment)).

I'm not sure how or who would be responsible for resolving this quandary at this point.

[marcpopMSFT]

Is this just in the standard runner image from GH actions? I think we need to set up a repro we can try a few things with as this definitely looks like a bad install. If this were a local machine, I could tell you how to resolve that. Also, I'm a bit curious how workload restore would work given that the SDK is installed into program files and restore would need permissions to update that. Is the build running as admin?

CC @baronfel @Forgind to discuss if we can easily set this up.

[baronfel]

@bdach can you set DOTNET_INSTALL_DIR to a custom location (as described here) and see if that makes your workload restore complete correctly? That will be a completely isolated file-based install (install of the weird hybrid msi/file-system based install that the default runners setup gets you) and should be a more isolated repro.

[bdach]

@marcpopMSFT:

Is this just in the standard runner image from GH actions?

Correct, that is the default windows runner image.

Also, I'm a bit curious how workload restore would work given that the SDK is installed into program files and restore would need permissions to update that. Is the build running as admin?

I'm not sure, sorry. I'd rather avoid researching this if I can.

@baronfel:

can you set DOTNET_INSTALL_DIR to a custom location (as described here) and see if that makes your workload restore complete correctly

Yes, this can be made to work with DOTNET_INSTALL_DIR. See updated variant of my minimal reproducer that passes. It's still a pretty suboptimal setup and requires going out of my way significantly (e.g. to check out the code to a separate directory, to ensure that the install doesn't get put in the repository root, or that the repository checkout doesn't clobber the local dotnet install).

You may also find it relevant that @thomasbach-dk reported in the actions/runner-images issue thread that installing .NET 7.0.x appears to somehow fix this, and I can corroborate. See relevant passing run.

The original failing variant of the reproducer can be found here.

[marcpopMSFT]

Sounds like you found a workaround. We think what's happening is that the runner infrastructure is installing file-based SDKs into the Admin-based location. This works much of the time but not actually supported. Given all the different versions of the SDK installed, there must be some combination of workloads between 9 and 6 that are not compatible but installing 7 short-circuits the fallback logic. We'll follow up with the runner folks on this to understand more.