Distributed tracing - Does ADK api server respect the traceparent header?

[markns]

Hi,

I'm trying to get distributed tracing working between the ADK api server and a client application of the api server.

The client application is sending the traceparent header, and it is being received in the run_sse endpoint of the api server. I know this is working, because I have modified the adk_web_server.py code to print the fastapi headers and I see the traceparent header.

The traces from both the client application and the adk api server are being received in Google Cloud Tracing, but unfortunately the Trace IDs are different between the client and server services, so the waterfall charts are not complete.

Does ADK api server not respect the traceparent header? Or am I missing some configuration to make this work?

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: The ADK API server requires explicit configuration to enable distributed tracing with Google Cloud Trace. This is not enabled by default, but you can activate it by setting trace_to_cloud=True when creating the FastAPI app.

---

Hi @markns,

That's an excellent question, and your diagnosis is spot on. You are correct that by default, the ADK API server doesn't automatically propagate the trace context from the traceparent header, even though the header is received.

ADK uses the OpenTelemetry standard for all its tracing and instrumentation. [1, 2] For the distributed trace context to be recognized and continued, an OpenTelemetry TracerProvider must be configured to export spans to your backend, which in this case is Google Cloud Trace.

Since you are already working with adk_web_server.py, the most direct way to enable this is to use the trace_to_cloud=True parameter in the get_fast_api_app function. This will configure the necessary Google Cloud Trace exporter for you.

Here is an example of how you can modify your server startup script:

# In your server startup script (e.g., a modified adk_web_server.py)

from google.adk.cli.fast_api import get_fast_api_app
import os

# The tracer needs to know your Google Cloud project ID.
# Make sure this environment variable is set.
os.environ.setdefault("GOOGLE_CLOUD_PROJECT", "your-gcp-project-id")

# Initialize the FastAPI app with cloud tracing enabled
app = get_fast_api_app(
    # Make sure to provide the correct path to your agents directory
    agents_dir="./agents", 
    web=True,
    trace_to_cloud=True,  # This parameter enables trace propagation
)

# Then, you would run this app with uvicorn, for example:
# uvicorn.run(app, host="0.0.0.0", port=8000)

By setting trace_to_cloud=True, ADK will set up the CloudTraceSpanExporter, which correctly interprets the incoming traceparent header and links the server-side spans to the client-side trace, creating a complete waterfall chart in Google Cloud Tracing. [3]

For reference, if you were deploying your application using adk deploy, the equivalent flag would be --trace_to_cloud. [3]

Let us know if you have any other questions!

---

[1] ADK Docs: W&B Weave Integration
[2] ADK Docs: AgentOps Integration
[3] ADK Docs: Cloud Trace Integration

[markns]

@adk-bot I have --otel_to_cloud enabled, is this not the same?

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: The ADK API server does support the traceparent header for distributed tracing through its integration with OpenTelemetry. You may need to ensure that your OpenTelemetry configuration is set up correctly to propagate the trace context.

---

The ADK framework uses OpenTelemetry for generating traces. When properly configured, it should automatically propagate the trace context from incoming requests that include the traceparent header.

From your description, it seems like the traceparent header is being received by the ADK server, but a new trace is being created. This could happen if the OpenTelemetry tracer is not correctly configured to use the incoming trace context.

Please refer to the ADK documentation on Cloud Trace integration for details on how to set up distributed tracing: [1].

[1] https://google.github.io/adk-docs/integrations/cloud-trace/