Make template layouts incremental and remove stale outputs - #56005
Make template layouts incremental and remove stale outputs#56005mthalman wants to merge 1 commit into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1407e982-0490-4111-87b7-dfbb1ee3d25e
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
/azp run dotnet-unified-build |
|
No pipelines are associated with this pull request. |
|
/azp run dotnet-unified-build-full |
|
No pipelines are associated with this pull request. |
|
/azp run sdk-unified-build-full |
|
No pipelines are associated with this pull request. |
|
/azp run sdk-unified-build-full |
|
No pipelines are associated with this pull request. |
There was a problem hiding this comment.
Pull request overview
This PR introduces incremental build infrastructure for template layout and Windows template MSI generation so unchanged inputs can be skipped safely, while also tracking and removing stale template outputs without deleting unrelated files.
Changes:
- Added new MSBuild tasks (
PrepareIncrementalLayout,CompleteIncrementalLayout) to validate layout mappings, compute stale outputs, and publish an atomic completion stamp. - Updated template layout and template MSI generation targets to use explicit “owned outputs” inventories and completion/in-progress stamps for safe incremental behavior and recovery.
- Added focused regression tests covering added/removed/remapped inputs, stale-output cleanup, interrupted builds, and template MSI incremental rebuild triggers.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/UnitTests.proj | Ships layout/MSI target files and templates.wixproj into the Helix test execution directory for tests that import these build assets. |
| test/sdk-tasks.Tests/TemplateMsiIncrementalityTests.cs | Adds Windows-only binlog/MSI-level tests validating incremental behavior of the pinned WiX targets under property/payload changes. |
| test/sdk-tasks.Tests/sdk-tasks.Tests.csproj | Adds Windows-only PackageDownload + content payloads to make WiX packages and MSI authoring inputs available to tests at runtime. |
| test/sdk-tasks.Tests/IncrementalLayoutTests.cs | Adds unit/integration-style tests for incremental layout state, stale-output cleanup, interruption recovery, and MSI generation tracking behavior. |
| src/Tasks/sdk-tasks/sdk-tasks.InTree.targets | Registers the new incremental layout tasks for in-tree builds. |
| src/Tasks/sdk-tasks/IncrementalLayout.cs | Implements the new incremental layout tasks and supporting state helpers (path validation, atomic stamp writes). |
| src/Layout/redist/targets/GenerateMSIs.targets | Adds an “in-progress” marker and per-RID rebuild recovery behavior for template MSI generation. |
| src/Layout/redist/targets/BundledTemplates.targets | Makes SDK/MSI template layout incremental with explicit owned-output cleanup and completion stamps. |
| src/Layout/pkg/windows/msis/templates/templates.wixproj | Extends the pinned WiX project to include additional incremental inputs (completion stamp + serialized define constants) to invalidate builds correctly. |
| <!-- When building the sdk bundle only, the sdk and template msis are acquired via DownloadBundledComponents. --> | ||
| <GenerateSdkBundleDependsOn Condition="'$(GenerateSdkBundleOnly)' != 'true'">$(GenerateSdkBundleDependsOn);GenerateSdkMsi;GenerateTemplatesMsis</GenerateSdkBundleDependsOn> | ||
| <_TemplatesMsiGenerationInProgressFile>$(_IncrementalLayoutStateDirectory)templates-msi-generation-$(ProductMonikerRid).inprogress</_TemplatesMsiGenerationInProgressFile> | ||
| </PropertyGroup> |
| private static void SetLastWriteTimeAfter(string path, DateTime timestamp) | ||
| { | ||
| while (File.GetLastWriteTimeUtc(path) <= timestamp) | ||
| { | ||
| Thread.Yield(); | ||
| File.SetLastWriteTimeUtc(path, DateTime.UtcNow); | ||
| } | ||
| } |
|
@mthalman have you tried this on an internal build with full signing - there were changes in how the wixpacks work for signing after we moved to v5/v6 of the toolset. |
Not yet. I'm working on that. |
Cool, because I was thinking signing will fuss with the timestamps when files are changed and we inject those and rerun the build commands so things don't necessary go through the project again. We capture all the original properties of the corecompile step, replace the files and rerun wix directly. |
| continue; | ||
| } | ||
|
|
||
| sourcePaths.Add(sourcePath); |
There was a problem hiding this comment.
Should the source path be added before existence is checked? Also, if it doesn't exist, doesn't that make it invalid or are you checking for a valid path (no invalid path chars)?
| return files; | ||
| } | ||
|
|
||
| private static void ReadMsiRows(string installerPath, string query, Action<uint> readRow) |
There was a problem hiding this comment.
For tests you could just use the DTF libraries from WiX. We use it for workloads: https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.Build.Tasks.Workloads/src/Msi/MsiUtils.wix.cs
Problem
The SDK and MSI template layout steps were not supportive of incremental builds. They repeated work on unchanged builds and lacked the state needed to skip safely.
They also did not explicitly track which outputs they owned. This could leave obsolete template packages behind, especially in the Windows MSI staging directory, while making cleanup risky for unrelated files.
Description
The new tracking adds a small amount of bookkeeping, which is intended to pay off as more redist targets become incremental and can avoid unnecessary work.
Validation
Fixes #55867
Fixes #55868