Add UseInstrumentation support to dotnet run for device testing

[rmarinho]

Summary

Add support for launching Android apps via adb shell am instrument instead of adb shell am start when running device tests. This enables proper wait-for-completion semantics for test execution scenarios.

Background

The dotnet test for mobile platforms spec (dotnet/maui#33117) defines how dotnet test will work with device test projects. A working POC (rmarinho/testfx#2) demonstrates two execution modes for Android:

1. Activity Mode (current): Uses adb shell am start to launch MainActivity
2. Instrumentation Mode (proposed): Uses adb shell am instrument -w for reliable test execution

The Instrumentation Mode provides:

• Proper wait-for-completion via -w flag
• Reliable exit code propagation via Instrumentation.Finish()
• Standard Android test execution pattern

Proposed Changes

Modify _AndroidComputeRunArguments target in [Microsoft.Android.Sdk.Application.targets](https://github.com/dotnet/android/blob/main/src/Xamarin.Android.Build. Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android. Sdk.Application.targets#L52-L79) to support a new UseInstrumentation property.

Current Implementation

<!-- When WaitForExit is false, use direct adb command (no logcat streaming) -->
<PropertyGroup Condition=" '$(WaitForExit)' == 'false' ">
  <RunCommand>$(_AdbToolPath)</RunCommand>
  <RunArguments>$(AdbTarget) shell am start -S -n "$(_AndroidPackage)/$(AndroidLaunchActivity)"</RunArguments>
</PropertyGroup>

Proposed Enhancement

<!-- New: When UseInstrumentation is true, launch via adb instrument -->
<PropertyGroup Condition=" '$(UseInstrumentation)' == 'true' ">
  <_AndroidRunPath Condition=" '$(_AndroidRunPath)' == '' ">$(MSBuildThisFileDirectory)..\tools\Microsoft.Android.Run.dll</_AndroidRunPath>
  <RunCommand>dotnet</RunCommand>
  <RunArguments>exec "$(_AndroidRunPath)" --adb "$(_AdbToolPath)" --package "$(_AndroidPackage)" --instrument "$(AndroidInstrumentationName)" --logcat-args "$(_AndroidRunLogcatArgs)" $(_AndroidRunExtraArgs)</RunArguments>
</PropertyGroup>

<!-- Alternative: Direct adb instrument without Microsoft. Android.Run wrapper -->
<PropertyGroup Condition=" '$(UseInstrumentation)' == 'true' AND '$(WaitForExit)' == 'false' ">
  <RunCommand>$(_AdbToolPath)</RunCommand>
  <RunArguments>$(AdbTarget) shell am instrument -w "$(_AndroidPackage)/$(AndroidInstrumentationName)"</RunArguments>
</PropertyGroup>

New MSBuild Properties

Property	Description	Default
$(UseInstrumentation)	When true, use adb instrument instead of adb am start	false
$(AndroidInstrumentationName)	Full instrumentation class name (e.g., myapp.TestInstrumentation)	(none, required when UseInstrumentation=true, if not provided try to find the first on the manifest or using the Instrumentation attribute )

Usage

# Current (Activity Mode)
dotnet run --project MyTests.csproj -f net10.0-android --device emulator-5554

# Proposed (Instrumentation Mode)
dotnet run --project MyTests.csproj -f net10.0-android --device emulator-5554 -p:UseInstrumentation=true -p:AndroidInstrumentationName=myapp.TestInstrumentation

Test Project Requirements

Projects using Instrumentation Mode need:

1. TestInstrumentation class:

[Instrumentation(Name = "myapp.TestInstrumentation")]
public class TestInstrumentation : Instrumentation
{
    public override void OnCreate(Bundle? arguments)
    {
        base.OnCreate(arguments);
        Start();
    }

    public override async void OnStart()
    {
        base.OnStart();
        int exitCode = 1;
        Bundle results = new Bundle();
        
        try
        {
            // Run tests via Microsoft. Testing.Platform
            exitCode = await MicrosoftTestingPlatformEntryPoint.Main(args);
            results. PutString("status", exitCode == 0 ? "SUCCESS" : "FAILURE");
        }
        finally
        {
            // Signal completion - adb instrument -w waits for this
            Finish(exitCode == 0 ? Result.Ok : Result. Canceled, results);
        }
    }
}

2. AndroidManifest.xml registration:

<instrumentation
  android:name="myapp.TestInstrumentation"
  android:targetPackage="com.mycompany.mytestapp"
  android: label="Test Instrumentation" />

Why Instrumentation Mode?

Aspect	Activity Mode	Instrumentation Mode
Completion detection	App termination (unreliable)	Finish() call (reliable)
Exit code	Java.Lang.JavaSystem.Exit()	Finish(Result, Bundle)
Wait semantics	Polling/timeout	-w flag blocks until done
Long-running tests	May timeout	Waits indefinitely
Standard pattern	Custom	Android test convention

Related

• Spec: dotnet/maui#33117 - dotnet test for Mobile Platforms
• POC: rmarinho/testfx#2 - Working implementation with both modes
• Related issue: dotnet/android#9496

/cc @jonathanpeppers @rmarinho

[jonathanpeppers]

One further improvement/question to this:

The $(UseInstrumentation) name sounds fine in isolation, but does dotnet test already pass in an MSBuild property to dotnet run? If not, should it? In theory, I could have an app that dotnet run should launch an activity and dotnet test runs the instrumentation. This would let you use a UI to run tests, or CLI, like we do in many test apps today.

[rmarinho]

We can taylor what properties are passed to dotnet run in our device test targets. MTP has this property UseMSBuildTestInfrastructure that when true tells MTP to let MSBuild infrastructure (SDK/custom targets) handle test execution instead of directly invoking the test executable. This is essential for device testing because we need dotnet run
--device to deploy and run on the device, not direct executable invocation.
So I think the user can just have its instrumentation file, and when he calls dotnet test, we create a dotnet run command with UseInstrumentation=true and run the tests.
If you want a UI to run tests, you can just implement it yourself on your app, use the TestApplicationBuilder from MTP and call RunAsync and capture yourself the results. We can provide samples for that in the future.
Here is an example of calling right now to run the tests from a MAUI app https://github.com/rmarinho/testfx/blob/add-device-tests/samples/public/TestingPlatformExamples/MauiApp1/MainPage.xaml.cs#L74

[jonathanpeppers]

What I'm asking, if we make a new property UseInstrumentation=true, then we have to change dotnet test to pass it.

If they are already passing a different property, can we use the existing one? (UseInstrumentation only makes sense as a name on Android)