AWS lambda deployments are not invoked with an AWS Xray trace ID header

[mupperton]

I'm using some TypeScript lambda deployments, with active tracing turned on in AWS, also using OTEL SDK and restate OTEL hook in the lambda function

I'm noticing that the AWS Lambda service is creating its own Trace ID because it appears to ignore the traceparent header that Restate Server sends in the invocation event, so the lambda function sees the environment variable _X_AMZN_TRACE_ID which has a differing trace ID to the traceparent

This leads to a "broken" trace, where I see one trace (in AWS traces) for the AWS side of invoking my function, and a separate trace for Restate Server and my deployment's service handlers, as the OTEL SDK in the deployment is using the W3C trace propagation and picking up the traceparent header in the event

I believe if the lambda invocation request from Restate Server was to also provide an additional header X-Amzn-Trace-Id with the right formatting, then AWS active tracing should adopt this and problem solved?
Also I believe there is a distinction between the header being in the event payload VS being an actual HTTP header in the request to the AWS platform, because I've tried manually invoking a lambda that has the Xray header name in the event, and no luck, whereas I've seen online that others are seeing results when they instrument their AWS-SDK with Xray and do a lambda-invoke that way, for example.

[mupperton]

JS example of how AWS format their header value for reference (copied from an AWS lib)

const otTraceId = spanContext.traceId;
const timestamp = otTraceId.substring(0, 8);
const randomNumber = otTraceId.substring(8);
const parentId = spanContext.spanId;
const samplingFlag = (TraceFlags.SAMPLED & spanContext.traceFlags) === TraceFlags.SAMPLED
    ? '1'
    : '0';
const traceHeader = `Root=1-${timestamp}-${randomNumber};Parent=${parentId};Sampled=${samplingFlag}`;