Use just-built packages in the AI chat template by default

[MackinnonBuck]

This PR makes the following changes to the chat template:

1. By default, reference just-built MEAI package versions
2. Use the Microsoft.EntityFrameworkCore.Sqlite version defined in Versions.props instead of being hard-coded
3. Add new MSBuild properties that can be specified to utilize public versions of template dependencies instead of just-built versions
4. Generate development-only files that allow the templates to utilize locally produced NuGet packages
5. Adapt to the recent breaking changes in MEAI.

Microsoft Reviewers: Open in CodeFlow

[jeffhandley]

@MackinnonBuck I merged from main, including the snapshot tests, and tried to work through getting everything green again. I've hit a couple of snags that we'll have to figure out though.

1. The snapshot is specific to whether we're using the pinned version or the just-built version, with the NuGet.config and Directory.Build.Targets files either included or excluded.
2. The snapshot bakes in the just-built version number, which is different between local development and CI.
3. Question: Is there something in place already where for the official build pipeline we would be able to toggle which mode to build with, or would we just update the targets file to toggle which is the default based on the next release's approach?

To get around the snapshot issues in CI and since we expect our next template release to use the MEAI and Sqlite packages released earlier this week, I swapped it to use the pinned versions by default, and I updated to the versions released this week. This got tests passing locally and in CI.

I hope you don't mind I pushed those changes on into your branch. Feel free to rewind the history and force-push if you want to discard my commits.

[MackinnonBuck]

Thanks for adding those updates, @jeffhandley!

1. The snapshot is specific to whether we're using the pinned version or the just-built version, with the NuGet.config and Directory.Build.Targets files either included or excluded.

Ideally the snapshot would ignore files that don't get included in the actual .nupkg, but the tests just treat the source files in the repo as the template content and ignore what gets specified as <Content> in Microsoft.Extensions.AI.Templates. We could try to figure out an alternative that somehow takes the build output of M.E.AI.Templates into account.

2. The snapshot bakes in the just-built version number, which is different between local development and CI.

Yeah, also unfortunate. If we did my suggestion above, maybe the tests could build M.E.AI.Templates and specify MSBuild props that normalize that version number somehow (maybe even using a fake one). Or, I think there's a way for the tests to do some post-processing on the output (here); maybe we could do the normalization there. In fact, maybe the template testing infrastructure allows us to completely remove files from the snapshot (like NuGet.config and Directory.Build.targets), which would also solve 1.).

3. Question: Is there something in place already where for the official build pipeline we would be able to toggle which mode to build with, or would we just update the targets file to toggle which is the default based on the next release's approach?

If the official build pipeline allows us to customize the build command on the fly, we could specify /p:TemplateUsePinnedMicrosoftExtensionsAIVersion or any of the other properties in GeneratedContent.targets. Otherwise we'd have to edit the .targets file manually. Maybe someone more familiar with that area of the build infrastructure would know better than I do. @RussKie?

[MackinnonBuck]

Updates:

• Snapshots now remove the -ci and -dev version suffixes
• Snapshots now ignore content that doesn't get included in the M.E.AI.Templates NuGet package
• The templates now utilize a new AddMessages() API

[RussKie]

@MackinnonBuck I merged from main, including the snapshot tests, and tried to work through getting everything green again. I've hit a couple of snags that we'll have to figure out though.

1. The snapshot is specific to whether we're using the pinned version or the just-built version, with the NuGet.config and Directory.Build.Targets files either included or excluded.
2. The snapshot bakes in the just-built version number, which is different between local development and CI.
3. Question: Is there something in place already where for the official build pipeline we would be able to toggle which mode to build with, or would we just update the targets file to toggle which is the default based on the next release's approach?

To get around the snapshot issues in CI and since we expect our next template release to use the MEAI and Sqlite packages released earlier this week, I swapped it to use the pinned versions by default, and I updated to the versions released this week. This got tests passing locally and in CI.

I hope you don't mind I pushed those changes on into your branch. Feel free to rewind the history and force-push if you want to discard my commits.

Thanks for adding those updates, @jeffhandley!

1. The snapshot is specific to whether we're using the pinned version or the just-built version, with the NuGet.config and Directory.Build.Targets files either included or excluded.

Ideally the snapshot would ignore files that don't get included in the actual .nupkg, but the tests just treat the source files in the repo as the template content and ignore what gets specified as <Content> in Microsoft.Extensions.AI.Templates. We could try to figure out an alternative that somehow takes the build output of M.E.AI.Templates into account.

2. The snapshot bakes in the just-built version number, which is different between local development and CI.

Yeah, also unfortunate. If we did my suggestion above, maybe the tests could build M.E.AI.Templates and specify MSBuild props that normalize that version number somehow (maybe even using a fake one). Or, I think there's a way for the tests to do some post-processing on the output (here); maybe we could do the normalization there. In fact, maybe the template testing infrastructure allows us to completely remove files from the snapshot (like NuGet.config and Directory.Build.targets), which would also solve 1.).

3. Question: Is there something in place already where for the official build pipeline we would be able to toggle which mode to build with, or would we just update the targets file to toggle which is the default based on the next release's approach?

If the official build pipeline allows us to customize the build command on the fly, we could specify /p:TemplateUsePinnedMicrosoftExtensionsAIVersion or any of the other properties in GeneratedContent.targets. Otherwise we'd have to edit the .targets file manually. Maybe someone more familiar with that area of the build infrastructure would know better than I do. @RussKie?

I think this discussion directly relates and is the cause of #6128.

Can you help me understand why the versions are different between local development and CI - or more specifically between the public and the internal CI?

[MackinnonBuck]

Can you help me understand why the versions are different between local development and CI - or more specifically between the public and the internal CI?

@RussKie, the reason is because the version suffix when building locally is -dev, but when building in CI it's -ci. I had some changes in this PR that removed the suffix from the snapshots as an attempt to "normalize" the version number. But it looks the internal branches use yet another suffix (-preview.x.x.x).

We could probably update the test to also scrub versions matching that suffix as well. Or we could do something possibly more robust and find the exact <PackageReference />s for packages defined within the repo and completely scrub the Version.