Error on site extension transform

[gkatz-flare]

Description

When setting up the site extension Datadog.AzureAppServices.DotNet for an Azure Function v4, the extension installs successfully, however there are errors on site transform and no metrics are sent to Datadog.

Context

Datadog.AzureAppServices.DotNet: v2.11.0
Azure Functions: v4
.NET: 6.0

Logging

%AzureAppServiceHomeDirectory%\LogFiles\datadog

[(v2.11.000) 2022-07-07 00:50 +00:00] Beginning install
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueStatsPipeId to ECCFB9EB-E296-4BB9-930E-88EFCA34A745 in .\applicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueTracePipeId to 7F8F4DAB-CD6F-493E-AAE1-DE0068B0F411 in .\applicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueStatsPipeId to ECCFB9EB-E296-4BB9-930E-88EFCA34A745 in .\scmApplicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Setting uniqueTracePipeId to 7F8F4DAB-CD6F-493E-AAE1-DE0068B0F411 in .\scmApplicationHost.xdt
[(v2.11.000) 2022-07-07 00:50 +00:00] Install complete

%AzureAppServiceHomeDirectory%\LogFiles\Transform

2022-07-07T01:00:30 sandboxproc.exe D:\DWASFiles\Sites#1fn-xxxx-api\Temp\applicationhost.config False True
2022-07-07T01:00:30 env XPROC_TYPENAME=Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper, Microsoft.Web.Hosting, Version=7.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
2022-07-07T01:00:30 env XPROC_METHODNAME=Transform
2022-07-07T01:00:31 Start 'Datadog.AzureAppServices.DotNet' site extension transform
2022-07-07T01:00:31 sandboxproc.exe Elapsed = 485.00 ms failed with System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Xml.XmlException: The 'configuration' start tag on line 2 position 2 does not match the end tag of 'runtime'. Line 25, position 7.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag)
at System.Xml.XmlTextReaderImpl.ParseEndElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlDocument.Load(XmlReader reader)
at Microsoft.Web.XmlTransform.XmlFileInfoDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at Microsoft.Web.XmlTransform.XmlTransformation..ctor(String transform, Boolean isTransformAFile, IXmlTransformationLogger logger)
at Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionDefinition.Transform(XmlDocument document, IDictionary2 environments, Boolean isMainSite, Boolean separateProcessForScm) at Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper.Transform(XmlDocument doc, List1 definitions, IDictionary`2 environments, Boolean isMainSite, Boolean separateProcessForScm)
at Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper.Transform(String appHostConfig, String isMainSiteStr, String separateProcessForScmStr)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.Web.Hosting.ProcessModel.Program.Main(String[] args)

DD_TRACE_DEBUG has been set to true, but no other logs are present.

Possible Solution

https://github.com/DataDog/datadog-aas-extension/blob/master/dotnet/content/scmApplicationHost.xdt contains a mismatched runtime tag, as referenced by the error. This seems to be fixed in #111 so it is possible that merging this PR will solve the issue.

Additional Information

I have an open ticket with Datadog here: https://help.datadoghq.com/hc/en-us/requests/855745

[pierotibou]

Hello, Thanks for raising the issue, we have also received the ticket through our support indeed.
I started to look into it, I don't think that the scm issue should prevent any tracing, it just prevents us from properly instrumenting webjobs.

I'll resume the testing on #111, to see if it can unblock you and add the same time add support for webjobs, meanwhile can you share a bit more details:

• I assume the extension files are present in /SiteExtensions
• IIUC, you don't have any log files in /logfiles/datadog/v{version}, right?
• Can you also check in process explorer, on the IIS process (not the scm one) if you have the following environment variables?
• Finally, can you tell me if you are using in-process or out-of-process functions?

Thanks in advance

[gkatz-flare]

Thanks for getting on this so quick! In regards to your questions:

• I assume the extension files are present in /SiteExtensions

Yep they are there

• IIUC, you don't have any log files in /logfiles/datadog/v{version}, right?

Yep there are no logs there. The version folder does not exist.

• Can you also check in process explorer, on the IIS process (not the scm one) if you hqve the following environment variables?

I cannot find any of those env variables on the process

• Finally, can you tell me if you are using in-process or out-of-process functions?

We are using out-of-process (isolated) functions

[gavkat]

@pierotibou - is there any update on this? Let me know if you need any more information.

[zacharycmontoya]

Hi @gavkat , we have merged #135 and it is included in the latest releases. Can you update your extension and let us know if you still run into issues?

[gkatz-flare]

Hi @zacharycmontoya , I am no longer seeing the error I described above, but I still cannot see any APM tracing in datadog for that function, and, even though DD_TRACE_DEBUG has been set to true, no other logs are present.

I also checked to see if anything changed regarding the questions @pierotibou asked, but the answers all remain the same. The site extension exists, there are no files in /logfiles/datadog/v{version}, and the environment variables listed are still not present.