ddtrace/tracer: add integration tag to spans_started/finished

[hannahkm]

What does this PR do?

Add a integration tag to the existing datadog.tracer.spans_started and datadog.tracer.spans_finished metrics. The value of the tag will be the name of the component from which the span was started. For example, for a contrib, it will be the name of the contrib package (chi, net/http, etc). For spans that were created manually, the tag will say manual.

For the purpose of adding tags, we move the logic of counting finished spans to span.finish() rather than trace.finishChunk(). Since we must read the integration data from each individual span, we would rather increment our counter each time one span finishes. Counting in finishChunk() as we did previously would require a for loop, which might impact efficiency.

Motivation

We want to know, in addition to when a span is started, where the span originated from. This could be a contrib or a manual implementation.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[datadog-datadog-prod-us1]

Datadog Report

Branch report: apm-rd/span-source-health-metric
Commit report: 2756596
Test service: dd-trace-go

✅ 0 Failed, 5211 Passed, 72 Skipped, 2m 46.57s Total Time

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-02-24 20:15:18

Comparing candidate commit 3ce582f in PR branch apm-rd/span-source-health-metric with baseline commit 386c125 in branch main.

Found 1 performance improvements and 9 performance regressions! Performance is the same for 51 metrics, 1 unstable metrics.

scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-24

• 🟥 execution_time [+196.132ns; +255.468ns] or [+3.991%; +5.198%]

scenario:BenchmarkOTelApiWithCustomTags/otel_api-24

• 🟥 execution_time [+187.474ns; +242.126ns] or [+2.447%; +3.160%]

scenario:BenchmarkPartialFlushing/Disabled-24

• 🟥 execution_time [+13.889ms; +16.283ms] or [+5.122%; +6.005%]

scenario:BenchmarkPartialFlushing/Enabled-24

• 🟥 execution_time [+15.173ms; +17.179ms] or [+5.476%; +6.200%]

scenario:BenchmarkSetTagStringer-24

• 🟩 execution_time [-6.041ns; -3.119ns] or [-4.128%; -2.131%]

scenario:BenchmarkSingleSpanRetention/no-rules-24

• 🟥 execution_time [+16.249µs; +17.158µs] or [+6.799%; +7.179%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24

• 🟥 execution_time [+15.194µs; +16.463µs] or [+6.311%; +6.838%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24

• 🟥 execution_time [+14.940µs; +16.378µs] or [+6.198%; +6.795%]

scenario:BenchmarkStartRequestSpan-24

• 🟥 execution_time [+23.195ns; +24.165ns] or [+5.215%; +5.433%]

scenario:BenchmarkStartSpan-24

• 🟥 execution_time [+74.902ns; +90.098ns] or [+3.307%; +3.978%]

[mtoffl01]

Ok, so you're reporting spansStarted/spansFinished on span.Start/span.Finished if the integration is not empty, and leaving the chunk reporting to any spans that are manual... I understand why you did this but not totally sure about the approach.

span.Start and span.Stop are typically called quite frequently, so if a majority of the spans are from automatic integrations, this will be very noisy (and defeats the purpose of reporting the metrics at a specified interval, to reduce noise)

One alternative idea:
Change the way we track spansStarted and spansFinished to be some kind of counter map that includes the integration name, e.g. map[string]uint32 where the key is the integration name and the value is the count of spans started/finished that integration name. Then, in this goroutine, we'll have to iterate over the map and report the spans started/finished per integration
(or some other idea I haven't thought of?)

[hannahkm]

@mtoffl01 Good points! A map would probably work better; I was hesitant at first since I didn't want to change too much of what already exists, but knowing that these metrics are pretty old... I'm more down to change it up now.

[darccio]

@hannahkm I'm approving this but we should investigate why the benchmarks report the increased allocations.

[mtoffl01]

Overall, I definitely have some concerns 🤔 Maybe you can write some additional tests to provide peace of mind....

1. Tests designed to try and make the system fail -- what happens when you have multiple goroutines access a start span / finish span method, can we prove that we've protected against a race condition?
2. Maybe you want to write dedicated benchmarks to show how much performance is impacted