`AdditionalRuntimesStrategy` loads a reference assembly from a runtime pack, throwing `BadImageFormatException` (x86)

[paulirwin]

Summary

When test code triggers an AssemblyLoadContext.Resolving event for an assembly that exists as a reference assembly in an additional runtime pack (e.g. System.Security.Permissions.dll in Microsoft.WindowsDesktop.App), AdditionalRuntimesStrategy.TryToResolve calls LoadFromAssemblyPath() on it unconditionally. Reference assemblies (Version=0.0.0.0) cannot be loaded for execution, so the load throws BadImageFormatException. Because the resolver does not catch it, the exception propagates out of whatever call raised the Resolving event (here a Type.GetType(...) call), often turning a benign "type not found → null" into a hard failure.

I observe this on x86 test runs (x64 resolves a real assembly and does not hit the reference copy).

Environment / versions

• NUnit Framework: 4.6.1 (the only framework version in use)
• NUnit Engine: nunit.engine.core, Version=3.22.0.0 (bundled inside NUnit3TestAdapter 6.2.0)
• NUnit3TestAdapter: 6.2.0
• Microsoft.NET.Test.Sdk: 17.11.1
• Install method: NuGet (PackageReference)
• Runner: dotnet test (VSTest + NUnit3TestAdapter). Not the standalone nunit-console.
• OS / arch: Windows; reproduces with dotnet test --arch x86, does not reproduce with --arch x64
• Target framework: net8.0
• Required precondition: a Microsoft.WindowsDesktop.App shared runtime is installed (reproduced with 3.1.32)

Full command line

dotnet test -c Release --arch x86      # fails
dotnet test -c Release --arch x64      # passes

Component

The failing types are NUnit.Engine.Internal.TestAssemblyResolver and AdditionalRuntimesStrategy, which live in nunit.engine.core (consumed here via NUnit3TestAdapter 6.2.0, which bundles nunit.engine.core, Version=3.22.0.0). The current main of this repo still contains the unguarded load in src/NUnitCommon/nunit.agent.core/TestAssemblyResolver.cs:

public override bool TryToResolve(AssemblyLoadContext loadContext, AssemblyName assemblyName, [NotNullWhen(true)] out Assembly? loadedAssembly)
{
    loadedAssembly = null;
    if (assemblyName.Version is null)
        return false;

    if (!DotNet.FindBestRuntime(assemblyName.Version, _runtimeName, _x86, out DotNet.RuntimeInfo? runtime))
        return false;

    string candidate = Path.Combine(runtime.Path, runtime.Version.ToString(), assemblyName.Name + ".dll");
    if (!File.Exists(candidate))
        return false;

    loadedAssembly = loadContext.LoadFromAssemblyPath(candidate);   // <-- no check that `candidate` is runnable; no catch
    return true;
}

There is no check that candidate is a runnable (non-reference) assembly, and the LoadFromAssemblyPath call is not guarded, so a reference assembly in the runtime pack directory takes the whole resolution down.

Minimal repro

Two files. Run with dotnet test --arch x86 (fails) and --arch x64 (passes).

Repro.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
    <PackageReference Include="NUnit" Version="4.6.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="6.2.0" />
  </ItemGroup>
</Project>

ReproTests.cs

using System;
using NUnit.Framework;

namespace Repro
{
    public class Tests
    {
        // System.Security.Policy.PolicyException only exists on .NET Framework, so on modern
        // .NET, Type.GetType(..., throwOnError: false) is documented to return null (never throw).
        // Under the NUnit host on x86 it instead throws BadImageFormatException.
        // Passes on x64; fails on x86.
        [Test]
        public void ProbeMustNotThrow()
        {
            Assert.DoesNotThrow(
                (TestDelegate)(() => Type.GetType("System.Security.Policy.PolicyException, mscorlib", throwOnError: false)));
        }
    }
}

Actual result (x86)

System.BadImageFormatException : Could not load file or assembly 'System.Security.Permissions, Version=0.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. Format of the executable (.exe) or library (.dll) is invalid.
  ----> System.BadImageFormatException : Could not load file or assembly 'C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App\3.1.32\System.Security.Permissions.dll'. Format of the executable (.exe) or library (.dll) is invalid.
   at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
   at NUnit.Engine.Internal.AdditionalRuntimesStrategy.TryToResolve(AssemblyLoadContext loadContext, AssemblyName assemblyName, Assembly& loadedAssembly)
   at NUnit.Engine.Internal.TestAssemblyResolver.OnResolving(AssemblyLoadContext loadContext, AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(...)
   at System.Reflection.RuntimeAssembly.GetTypeCore(...)
   at System.Type.GetType(String typeName, Boolean throwOnError)
   at Repro.Tests.ProbeMustNotThrow()

Failed: 1 on x86; Passed: 1 on x64.

Expected result

Type.GetType(..., throwOnError: false) for a type that does not exist on the current runtime should return null. The resolver should not turn a benign resolution miss into a BadImageFormatException by attempting to execute a reference assembly.

Suggested fix

In AdditionalRuntimesStrategy.TryToResolve, either skip reference assemblies or guard the load and treat a failure as "not resolved":

try
{
    loadedAssembly = loadContext.LoadFromAssemblyPath(candidate);
    return true;
}
catch (BadImageFormatException)
{
    // candidate is a reference assembly (or otherwise not runnable); treat as unresolved
    loadedAssembly = null;
    return false;
}

Notes / related

• Distinct from Wrong assembly resolving logic when several runtimes are installed and assembly is missing in deps.json #1330 (wrong version selection) and TypeInitializationException in TestAssemblyResolver Targeting .NET 10 #1779 (ArgumentNullException/path1 in TestAssemblyResolver); both are closed and neither addresses loading a reference assembly.
• Consumer-side workaround for anyone hitting this from their own Type.GetType(..., "...mscorlib") probes: wrap the call and swallow BadImageFormatException/FileLoadException, returning null.