Publish Microsoft.Testing.Platform coverage messages from coverlet.MTP

[Evangelink]

Context

coverlet.MTP currently generates coverage reports from its ITestHostProcessLifetimeHandler. Another MTP extension that needs those reports cannot safely use the same callback because registration order between independent extensions is not guaranteed. This is the race reported in microsoft/testfx#8904 for ReportGenerator.Mtp.

Microsoft.Testing.Platform has now merged a first-class coverage message and consumer model in microsoft/testfx#9896, planned for the currently unreleased MTP 2.4.0:

• TestCoverageReportMessage publishes the path and format of a generated report.
• TestCoverageMessage publishes covered/coverable counts by scope and metric.
• TestCoverageThresholdMessage publishes threshold evaluations.
• ITestCoverageResult exposes the correlated reports, measurements, and thresholds to consumers.
• ITestCoverageCapabilities lets consumers detect enabled coverage producers.

Request

Once coverlet.MTP can target MTP 2.4.0, could it adopt these contracts?

At minimum, CollectorExtension could implement IDataProducer, advertise TestCoverageReportMessage in DataTypesProduced, and publish one message on IMessageBus after each on-disk coverage report is successfully written. Publishing TestCoverageMessage for Coverlet's summary metrics and TestCoverageThresholdMessage for its threshold results would also let MTP's built-in terminal and CI reporters consume Coverlet data consistently.

ReportGenerator.Mtp could then consume the report references through ITestCoverageResult during ITestSessionLifetimeHandler.OnTestSessionFinishingAsync (or consume TestCoverageReportMessage directly), removing the cross-extension lifetime-handler ordering dependency and file-glob polling.

Relevant design: RFC 019 — Code coverage messages & consumer model.

[Bertk]

@Evangelink The current coverlet.MTP implementation already supports many aspects, and draft PR #2019 adds failure threshold functionality.

Going forward, using Microsoft.Testing.Platform coverage messages could replace some parts, but there are also gaps — for example, terminal output in TeamCity format. This is used by CI systems and has been requested by coverlet users (see PR #1931).

Generally, coverlet packages use the data structure from coverlet.core/Coverage.cs and generate reports supporting multiple formats. As I understand it, the MTP module code coverage message should be sent per module with line numbers hits (not %), but this part doesn't exist yet and needs further analysis.

I will start this as soon as MTP 2.4.0 is released. Unfortunately, TeamCity format is currently not supported by MTP coverage messages, and we cannot drop it, as that would be a breaking change.

[Evangelink]

Thanks for the details. I checked the current MTP coverage contract against Coverlet's data model and TeamCity reporter.

The per-module data is already covered: TestCoverageMessage carries counts (not percentages) for any CoverageScope, including CoverageScopeLevel.Module. Coverlet can publish Line, Branch, and Method messages for each module using the existing CoverageDetails.Covered and CoverageDetails.Total values, and can also publish the corresponding Overall messages.

TeamCity is different from the formats represented by TestCoverageReportMessage: it is a console service-message protocol, not an on-disk coverage artifact. We therefore should not add CoverageReportFormat.TeamCity or a raw console-output coverage message. Coverlet can keep emitting its current TeamCity output through IOutputDevice alongside the new coverage messages, so adopting the MTP contracts does not require dropping TeamCity support or making a breaking change.

If MTP owns TeamCity rendering in the future, the clean design would be a separate TeamCity-specific consumer over ITestCoverageResult, emitting the six CodeCoverageAbs* service messages from the overall Line, Branch, and Method counts. The required coverage read model and output-device abstraction already exist.

One unrelated file-format gap is Coverlet JSON. It can be published today as CoverageReportFormat.Custom with customFormatName: "json"; we can consider a dedicated well-known enum member before the API ships if there is a concrete cross-tool consumer need.

[Bertk]

TeamCity is different from the formats represented by TestCoverageReportMessage: it is a console service-message protocol, not an on-disk coverage artifact. We therefore should not add CoverageReportFormat.TeamCity or a raw console-output coverage message. Coverlet can keep emitting its current TeamCity output through IOutputDevice alongside the new coverage messages, so adopting the MTP contracts does not require dropping TeamCity support or making a breaking change.

I agree — coverlet.MTP should publish TeamCity format results via console output.

One unrelated file-format gap is Coverlet JSON. It can be published today as CoverageReportFormat.Custom with customFormatName: "json"; we can consider a dedicated well-known enum member before the API ships if there is a concrete cross-tool consumer need.

Coverlet uses the JSON format internally and derives other report formats from it. The JSON format is also used to merge coverage reports. I'm not aware of any external data consumers, though some users may have built helper tools for CI. Using CoverageReportFormat.Custom should work, but I'd prefer customFormatName: "coverlet.json".

When will Microsoft.Testing.Platform v2.4.0 be released?

[Evangelink]

When will Microsoft.Testing.Platform v2.4.0 be released?

This week, hopefully today or tomorrow.