[Breaking change]: dotnet test removes dependency on Newtonsoft.Json

[nohwnd]

Description

dotnet test, Microsoft.NET.Test.SDK and the whole VSTest platform is no longer depending on Newtonsoft.Json. System.Text.Json is used instead on .NET and JSONite on .NET Framework.

Version

.NET 11 Preview 4

Previous behavior

In previous versions Newtonsoft.Json was used for communication between processes in VSTest. The dll was available in output folder, picked up by MSTest test adapter, and was referenced directly by .NET projects, through Microsoft.NET.Test.SDK nuget package..

This made the Newtonsoft.Json.dll available to projects that depended on it but for some reason did not bring it, and it made it available for compilation in .NET projects.

New behavior

Newtonsoft.Json is no longer available during compilation and runtime.

Type of breaking change

• Binary incompatible: Existing binaries might encounter a breaking change in behavior, such as failure to load or execute, and if so, require recompilation.
• Source incompatible: When recompiled using the new SDK or component or to target the new runtime, existing source code might require source changes to compile successfully.
• Behavioral change: Existing binaries might behave differently at run time.

Reason for change

Align dependencies with .NET SDK and the rest of the .NET ecosystem. Reduce the number of dependencies that test platform brings to output folder, to minimize the impact on tested code. Tested code is free to bring any dependencies it needs, and they should stay exactly the same version.

Recommended action

Test projects fail to compile

In .NET, Newtonsoft.Json is a compile time dependency of the test project. The test project may use it for serialization and deserialization in the test code, without installing the dependency themselves. Such a project would fail to compile after updating to version that no longer depends on Newtonsoft.Json.
This is a very visible error that will fail locally an in CI pipeline.

Mitigation

Users must install Newtonsoft.Json nuget in their project.

Tests fail with FileNotFoundException: Could not load 'Newtonsoft.Json'

Projects that build in non-standard way, may depend on the copy of Newtonsoft.Json we ship in with VSTest.
After VSTest migration to System.Text.Json, such code will fail to resolve the dependency and fail tests with standard assembly resolve error:

 FileNotFoundException: Could not load 'Newtonsoft.Json'

This will fail all affected tests, and fail the test run. This is a standard test method failure message, that will go into TRX and all the way up to Test view in AzDO / GitHub.
This is not a silent crash.
Who is affected

Users that explicitly exclude the dll from build, e.g. by:

<PackageReference Include="Newtonsoft.Json" Version="13.0.3">
  <ExcludeAssets>runtime</ExcludeAssets>
</PackageReference>

Mitigation

Users must install Newtonsoft.Json nuget in their project.

Test extensions fail with FileNotFoundException: Could not load 'Newtonsoft.Json'

Test extensions, such as test adapters or data collectors that relied on Newtonsoft.Json being shipped by VSTest and that did not claim it in their dependencies, would fail with

Data collector 'SampleDataCollector' message: Data collector 'SampleDataCollector'
threw an exception during type loading, construction, or initialization:
 System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, 
 Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.
The system cannot find the file specified. 

This error will fail the test run, and correctly identifies the extension that failed and that needs to be fixed (in this case SampleDataCollector).

Who is affected?

Currently there are no collectors or adapters we know of that are affected.

Mitigation

Users must install Newtonsoft.Json nuget in their project.
User can stop using affected Data Collector until its author fixes it.

Change of public API

VSTest currently depends on Newtonsoft.Json for its communication, and exposes Newtonsoft.Json type (JToken) in public API. This public API is used for communication and it is unlikely users depend on it.

We will remove the type from public API.

Change of data serializer

Communication between VSTest processes is done over TCP/IP with JSON serialized messages. We ensured that all messages serialize the same way when Newtonsoft.Json, System.Text.Json, or JSONite are used. We ensure both the structure and performance are the same or better.

We also ensure that we can still communicate with older versions of testhost to ensure backwards compatibility. It is no surprise that this works, the messages are the same, and there is airgap between the processes, so they have no idea what serializer the other side uses. Nor do they care if the message string remains the same.

Feature area

Other (please put exact area in description textbox)

Affected APIs

Newtonsoft.Json is no longer referenced by Microsoft.NET.Test.SDK, which makes it unavailable for compilation in .NET test projects.

Removed APIs that are used for communication between processes from Microsoft.VisualStudio.TestPlatform.CommunicationUtilities. This library is used by various components, but we don't ship it separately, and don't expect users to depend on it directly.

Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Message.Payload.get -> Newtonsoft.Json.Linq.JToken?
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Message.Payload.set -> void

Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.DefaultTestPlatformContractResolver
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.DefaultTestPlatformContractResolver.DefaultTestPlatformContractResolver() -> void
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestCaseConverter
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestCaseConverter.TestCaseConverter() -> void
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter.TestObjectConverter() -> void
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestPlatformContractResolver1
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestPlatformContractResolver1.TestPlatformContractResolver1() -> void
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestResultConverter
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestResultConverter.TestResultConverter() -> void
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestRunStatisticsConverter
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestRunStatisticsConverter.TestRunStatisticsConverter() -> void

Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.VersionedMessage
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.VersionedMessage.Version.get -> int
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.VersionedMessage.Version.set -> void
Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.VersionedMessage.VersionedMessage() -> void

override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.DefaultTestPlatformContractResolver.CreateContract(System.Type! objectType) -> Newtonsoft.Json.Serialization.JsonContract!
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestCaseConverter.CanConvert(System.Type! objectType) -> bool
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestCaseConverter.ReadJson(Newtonsoft.Json.JsonReader! reader, System.Type! objectType, object? existingValue, Newtonsoft.Json.JsonSerializer! serializer) -> object?
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestCaseConverter.WriteJson(Newtonsoft.Json.JsonWriter! writer, object? value, Newtonsoft.Json.JsonSerializer! serializer) -> void
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter.CanConvert(System.Type! objectType) -> bool
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter.CanRead.get -> bool
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter.CanWrite.get -> bool
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter.ReadJson(Newtonsoft.Json.JsonReader! reader, System.Type! objectType, object? existingValue, Newtonsoft.Json.JsonSerializer! serializer) -> object!
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestObjectConverter.WriteJson(Newtonsoft.Json.JsonWriter! writer, object? value, Newtonsoft.Json.JsonSerializer! serializer) -> void
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestPlatformContractResolver1.CreateContract(System.Type! objectType) -> Newtonsoft.Json.Serialization.JsonContract!
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestResultConverter.CanConvert(System.Type! objectType) -> bool
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestResultConverter.ReadJson(Newtonsoft.Json.JsonReader! reader, System.Type! objectType, object? existingValue, Newtonsoft.Json.JsonSerializer! serializer) -> object!
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestResultConverter.WriteJson(Newtonsoft.Json.JsonWriter! writer, object? value, Newtonsoft.Json.JsonSerializer! serializer) -> void
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestRunStatisticsConverter.CanConvert(System.Type! objectType) -> bool
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestRunStatisticsConverter.ReadJson(Newtonsoft.Json.JsonReader! reader, System.Type! objectType, object? existingValue, Newtonsoft.Json.JsonSerializer! serializer) -> object?
override Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.Serialization.TestRunStatisticsConverter.WriteJson(Newtonsoft.Json.JsonWriter! writer, object? value, Newtonsoft.Json.JsonSerializer! serializer) -> void

---

Associated WorkItem - 570494

[nohwnd]

Implemented via : dotnet/dotnet#6095

[nohwnd]

It was approved by .net tactics. Additional information can be found also in:

microsoft/vstest#15677

https://microsofteur.sharepoint.com/:fl:/g/contentstorage/x8FNO-xtskuCRX2_fMTHLQ9bj8F6uZZBpsF-If5XvhY/IQBhAm3iHM1tQIJElXLr4BezAb9DN4clytxD3u3E3fOXO8I?e=KX8KGc&nav=cz0lMkZjb250ZW50c3RvcmFnZSUyRng4Rk5PLXh0c2t1Q1JYMl9mTVRITFE5Ymo4RjZ1WlpCcHNGLUlmNVh2aFkmZD1iJTIxZXhyaEFfY09oa3VZZjdkS3JOMmlBcVpGTW1ITDYwSkFwNHZLdVZjYmdjb1NrTXhGazNxWVRyeFFYUEx0bVNCMyZmPTAxVzYzRTZDREJBSlc2RUhHTk5WQUlFUkVWT0xWNkFGNVQmYz0lMkYmYT1Mb29wQXBwJnA9JTQwZmx1aWR4JTJGbG9vcC1wYWdlLWNvbnRhaW5lcg%3D%3D (internal link)