Datadog AAS not tracing Azure ServiceBus events

[seif-fmfx]

We have the Azure App Service Extensions for Windows App Service, it is working great for http requests and sql server and I can see the logs linked to traces in datadog.

The application uses Paramaore.Brighter and Azure ServiceBus both of which publish events using an ActivitySource. Reading the docs for datadog, it looks like this should work.

There is some infrastructure code that adds the variables to to the app service webapp settings

DD_AAS_SCRIPT_INSTALL=1
DD_API_KEY=REDACTED
DD_ENV=dev
DD_SERVICE=MyService
DD_VERSION=1.0.0
DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS=Serilog
DD_LOGS_INJECTION=true
DD_PROCESS_AGENT_ENABLED=true
DD_RUNTIME_METRICS_ENABLED=true
DD_TRACE_CLIENT_IP_ENABLED=true
DD_SITE=datadoghq.eu

but I didn't want to add new variables to that before verifying that it works so I've set DD_TRACE_ENABLED and DD_TRACE_OTEL_ENABLED to true in the application's appsettings.json file and enabled the azure servicebus activity source with AppContext.SetSwitch("Azure.Experimental.EnableActivitySource", true); but don't see any traces being output.

I tried adding the dd-trace-dotnet as the docs indicate we can use that to push trace data to the agent (which if I understand correctly is what the extension bundles). Still not automatic ServiceBus traces were pushed.

I ended up implementing my own ActivityListener that sends traces to datadog:

 activityListener = new ActivityListener()
        {
            ShouldListenTo = a => true, 
            ActivityStopped = activity =>
            {
                var scope = tracer.StartActive(activity.OperationName,
                    new SpanCreationSettings()
                    {
                        StartTime = activity.StartTimeUtc,
                        FinishOnClose = true,
                    });                
                scope.Close();
            },
        };
        ActivitySource.AddActivityListener(activityListener);

but for some reason I wasn't getting the events I expected. The only way I got the events was by hooking up the OpenTelemetry library (it must be doing something that activates the sources as AcitivityListener). I don't have any exporters configured on the opentelemetry library:

        builder.Services.AddOpenTelemetry()
            .WithTracing(tracerProviderBuilder =>
                tracerProviderBuilder
                    .AddSource(
                        "Paramore.*",
                        "Microsoft.*",
                        "Azure.Messaging.*",
                        "Azure.Storage.*"));

I can now see traces in datadog for AzureServiceBusSender.Send, BlobUpload and so on but the traces are not linked and have different traceIds, I assume that it is because I am not setting the parent when starting the datadog activity. How would I go about converting the Hex String TraceId and SpanId into ulong so that I can create a Parent SpanContext?

Another thing I tried was using the OpenTelemetry exporter to export to the agent but for that to work I need to enable the OTLP ingest on the agents which I couldn't see how to do with the extension.

What is the best way to get traces from the azure services? It seems whatever I try, I get blocked and I couldn't find documentation about how to enable tracing those activities with the Extension.

I am not sure setting DD_TRACE_OTEL_ENABLED in the appsettings.json file is making any difference. Should that work or does it need to be an env variable? IIRC docs said it can be set in the config

We are using version 3.20.0 of the extension and dd-trace library. Any guidance/help would be appreciated.