[BUG]:Traces Dropped After 5 Retries When Sending to Local Agent Proxy

[ryuichi1208]

Tracer Version(s)

2.19.2

Python Version(s)

Python 3.11

Pip Version(s)

pip 22.04

Bug Report

When using version 2.19.2, the following error message occurs:

failed to send, dropping 1 traces to intake at http://localhost:8126/evp_proxy/v2/api/v2/llmobs

This issue arises in our application, which is a WebSocket-based service built with gunicorn and uses uvicorn as its worker. The traces fail to be sent to the local agent proxy (http://localhost:8126) after 5 retry attempts, leading to the traces being dropped. However, this issue is limited to traces sent to LLMObs, as traces to other destinations were successfully sent without any issues.

This issue does not occur in versions 2.18.2 or 2.18.0. If you need any additional information to investigate the problem, please let us know. We are happy to provide further details or logs as required.

Operating System

AWS Fargate for Amazon ECS

[adilnaimi]

I have the same issue as well:
Flask socketio application (flask-socketio)
gunicorn + eventlet
Deployed to ECS Fargate
ddtrace==2.19.0

failed to send, dropping 2 traces to intake at http://localhost:8126/evp_proxy/v2/api/v2/llmobs after 5 retries

[Yun-Kim]

Hi @ryuichi1208, thanks for reaching out and apologies for the delayed response.
Can you try upgrading to the latest ddtrace==2.20.0 or ddtrace==2.19.3 and see if this issue persists? I believe this may be related/fixed by #11961 which was merged after ddtrace==2.19.2.
If the issue persists, can you send over your tracer debug logs (DD_TRACE_DEBUG=1) to help us debug the underlying issue? Thanks!

[blimmer]

We are getting this on ddtrace==2.20.0 as well. We hadn't seen this on 2.18.0, our previous version.

[Yun-Kim]

@ryuichi1208 @blimmer @adilnaimi We're currently looking into this issue. In the meantime, would you be able to provide some more information to help us reproduce this? Thanks!

• What is your application setup (env vars, app startup command)?
• Which version of the Datadog agent are you running?
• Can you attach tracer debug logs (set DD_TRACE_DEBUG=1)?

[Amelioratede]

Hi, @Yun-Kim, I am also having this problem using 2.20.0 version which I resolved by switching to 2.21.1

But I still seeing some error log for connection error.

Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket 2025-02-20 15:19:53,690 - ddtrace.vendor.dogstatsd - WARNING - Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket 2025-02-20 15:20:05,013 - ddtrace.vendor.dogstatsd - WARNING - Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket

Do you have any idea for this?

[Yun-Kim]

@ryuichi1208 @blimmer @adilnaimi can you also confirm if upgrading to 2.21.1 resolves this issue?

@Amelioratede would you mind opening a new issue since this seems unrelated? Thanks!