Tracing not working in a weird scenario

[Jonathan-Chang]

Hello, I am very stuck on this. I am using aws-lambda-powertools version 2.16.2

I am using the decorator, capture_lambda_handler() in my lambda.

The flow of events goes like this:

SNS -> SQS -> Lambda w/ decorator -> other services

In this flow, I expect the trace to start at the lambda but nothing gets traced.

The cloudwatch logs shows no mention of it being traced.

However, if I publish a message directly into the SQS queue, tracing works.

I see within the cloudwatch logs of my lambda: xray trace id, segmentid, sampled: true

So putting a message directly into SQS through the console activates tracing but SNS publishing a message to SQS to then get picked up by my lambda doesn't.

Any ideas?

I just want to start tracing from my Lambda forward.

[leandrodamascena]

Hi @Jonathan-Chang! Maybe I need more information to try to help you. I created a scenario here and could have instrumented the Lambda with Powertools Tracer (using the capture_lambda_handler decorator) and have the tracing starting from Lambda onwards.

Steps I did:

1 - I created an SNS Topic with an SQS Subscription and Active Tracking disabled.
2 - In SQS I created a Lambda Trigger to invoke my Lambda
3 - In my Lambda I have the following code

from aws_lambda_powertools import Tracer
from aws_lambda_powertools.utilities.typing import LambdaContext

import boto3

ec2_cli = boto3.client("ec2")

tracer = Tracer(service="xyz")

@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> str:
    response = ec2_cli.describe_instances()
    return response

4 - I dropped a message either in SNS or SQS and the result is the same:

Can you check if you drop a message on SNS, does it get sent to SQS? I'm wondering here if it is missing setting up the Access Resource Policy in SQS to allow the SNS topic to send a message to SQS.

Please send me more information to try to help you resolve this tracing issue.

[Jonathan-Chang]

Hello, thank you for the response.

Yes, when a message gets dropped in SNS, it does get sent to SQS, the lambda picks it up and the lambda runs. It just doesn't get traced.

I should specify that it's a cross account SNS to SQS

Other account SNS -> my SQS -> my Lambda

I don't expect you to replicate that but maybe it's a potential factor.

I did create a subscription to a topic that exists in my account and dropped a message into my SNS topic, it got picked up by the SQS queue and then had Sampled: true in the cloudwatch logs of the lambda so that worked

This tracing worked in the past but suddenly stopped on April 27th around 3 PM.

I also don't have any specific sampling rules other than the default.

My execution role has the necessary permissions as well.

[heitorlessa]

but suddenly stopped on April 27th around 3PM

I'd definitely advise raising a support ticket as they have better tools to investigate that. If it never worked before, then we could try to replicate and check the usual suspects like Active tracing in SNS, whether X-Ray Console is able to see sources account linked, whether CloudWatch Unified Account is setup, etc.

[heitorlessa]

@Jonathan-Chang forgot to add in my response: please feel free to reach us out if you hit a dead end somehow.

https://github.com/aws-powertools/powertools-lambda-python#connect

[Jonathan-Chang]

I appreciate it @heitorlessa

Some information that support gave me that may be good to know is:

They had me look at the trace headers from the SQS queue that my Lambda picks up a message from.

Since it had Sampled=0; and Lineage in it, it means that the initial request that was sent to the SNS was done by a Lambda and this Lambda apparently doesn't have Active Tracing on which causes all the downstream to not have tracing on.

So I would need to alert the cross account Lambda to turn on their active tracing. I haven't confirmed this or tested this yet but this is what I got.

They are working to fix this issue but no ETA on it.

Thank you.