Issue in converting between OTEL and APM context

[rubvs]

Describe the bug

Information seems to be getting lost between converting from an APM context to OTEL context.

1. If we have a OTEL parent span and APM child span, we can see full relationship in Kibana. The figure below shows how we go from elasticHandler (otel) -> Streaming (apm) -> Semaphore.Acquire (apm)

2. The issue arises when we want to go from an APM span back to an OTEL span. In this case, we do not observe the last span (Semaphore.Acquire) in Kibana.

To Reproduce

1. Create an OTEL parent span. Then an APM child span, followed by another APM child span. View in Kibana
2. Create an OTEL parent span. Then an APM child span, followed by an OTEL child span. View in Kibana

Expected behavior

Spans and Span Links that go from OTEL context to APM context doesn't work.

[dmathieu]

Nice find.
It looks like this is because the APM span is a child span. We only look for transactions when checking parent spans from apm.

apm-agent-go/module/apmotel/tracer.go

Lines 77 to 79 in 2b38f4b

	if tx := apm.TransactionFromContext(ctx); tx != nil {
	parent = &span{tx: tx}
	}

[dmathieu]

My bad. We actually connect the span when it's created within the apm agent, through the wrapper.

apm-agent-go/module/apmotel/wrapper.go

Line 45 in 2b38f4b

func contextWithSpan(ctx context.Context, apmSpan *apm.Span) context.Context {

How are you creating the Stream span? Are you assigning it to the context?

[dmathieu]

@rubvs ping?

[rubvs]

@dmathieu Sorry, I was on PTO for 2 weeks. Here is how the Stream span is created, see Code. The context passed to this span is from an OTEL parent span.

[dmathieu]

I'm not seeing apmotel being required in apm-data (nor the service using it).
In order to be able to properly link spans together, apmotel need to be imported into the application, and the tracer provider needs to be configured, as is documented here: https://www.elastic.co/guide/en/apm/agent/go/current/opentelemetry.html