You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The google_adk LLM Observability integration intermittently drops the agent/request span entirely, emitting this error and leaving the child google_genai.request LLM spans orphaned (their parent_id points at a span that no longer exists in LLM Obs):
ERROR ddtrace.llmobs._llmobs: Error preparing LLMObs span event for span
<Span(id=...,name=google_adk.request)>, missing span kind in span context.
Root cause is an ordering bug in GoogleAdkIntegration._llmobs_set_tags. The span-kind annotation is the last statement, and it runs only if the preceding operation-specific extractor returns normally:
try:
self._llmobs_set_tags(span, args, kwargs, response, operation)
exceptException:
log.error("Error extracting LLMObs fields for span %s, likely due to malformed data", span, ...)
So if _llmobs_set_tags_agent (or _tag_agent_manifest) raises, the _annotate_llmobs_span_data(kind=...) line never executes → the span finishes with no kind → _prepare_llmobs_span_data rejects it and the span is dropped:
Fragile ordering — kind=operation should be committed before the fallible operation-specific extraction (or in a finally), so a failure in input/output/manifest extraction degrades gracefully (empty I/O) instead of dropping the whole span and orphaning its children.
The thing that raises is the same Gemini Part handling gap tracked in [BUG]: LLM Obs does not correctly handle google FilePart / ImagePart type #15720 ("LLM Obs does not correctly handle google FilePart / ImagePart type", closed not_planned by the stale bot, still unfixed on main). extract_message_from_part_google_genai has no branch for inline_data / file_data / empty / thought-signature parts and falls through to:
In the google_genai LLM-span path that fallback merely produces a confusing Unsupported file type: <class 'google.genai.types.Part'> string in the output (the #15720 symptom). In the google_adk agent-span path, the same malformed parts can take down the entire _llmobs_set_tags_agent extraction, which is what triggers the span drop described here.
We observe this with gemini-3.5-flash returning a normal text answer plus an extra empty / thought-signature Part — no images involved — so the trigger is broader than the image/file OCR case in #15720.
Reproduction Code
Run a google.adk agent whose model (gemini-3.5-flash) returns a response whose content.parts includes a part not handled by extract_message_from_part_google_genai (empty part, thought-signature part, or inline_data/file_data part), with LLM Obs enabled. The google_adk.request agent span is dropped and its child google_genai.request spans are orphaned.
Error Logs
Error preparing LLMObs span event for span <Span(id=14479202674267337881,name=google_adk.request)>, missing span kind in span context.
(Expect a companion Error extracting LLMObs fields for span ... likely due to malformed data immediately before it.)
Libraries in Use
google-adk 2.2.0
google-genai 2.8.0
Operating System
Linux (AWS ECS Fargate); also reproduces on Darwin 24.x
Tracer Version(s)
4.10.5 (also reproduces on
main)Python Version(s)
3.12.13
Pip Version(s)
uv 0.x / pip 24.x
Bug Report
The
google_adkLLM Observability integration intermittently drops the agent/request span entirely, emitting this error and leaving the childgoogle_genai.requestLLM spans orphaned (theirparent_idpoints at a span that no longer exists in LLM Obs):Root cause is an ordering bug in
GoogleAdkIntegration._llmobs_set_tags. The span-kind annotation is the last statement, and it runs only if the preceding operation-specific extractor returns normally:https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_integrations/google_adk.py#L30-L50
The caller swallows-and-logs any exception from
_llmobs_set_tags:https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_integrations/base.py#L114-L119
So if
_llmobs_set_tags_agent(or_tag_agent_manifest) raises, the_annotate_llmobs_span_data(kind=...)line never executes → the span finishes with no kind →_prepare_llmobs_span_datarejects it and the span is dropped:https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_llmobs.py#L614-L619
Two independent defects feed this:
Fragile ordering —
kind=operationshould be committed before the fallible operation-specific extraction (or in afinally), so a failure in input/output/manifest extraction degrades gracefully (empty I/O) instead of dropping the whole span and orphaning its children.The thing that raises is the same Gemini
Parthandling gap tracked in [BUG]: LLM Obs does not correctly handle google FilePart / ImagePart type #15720 ("LLM Obs does not correctly handle google FilePart / ImagePart type", closednot_plannedby the stale bot, still unfixed onmain).extract_message_from_part_google_genaihas no branch forinline_data/file_data/ empty / thought-signature parts and falls through to:https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_integrations/google_utils.py#L238
In the
google_genaiLLM-span path that fallback merely produces a confusingUnsupported file type: <class 'google.genai.types.Part'>string in the output (the #15720 symptom). In thegoogle_adkagent-span path, the same malformed parts can take down the entire_llmobs_set_tags_agentextraction, which is what triggers the span drop described here.We observe this with
gemini-3.5-flashreturning a normal text answer plus an extra empty / thought-signaturePart— no images involved — so the trigger is broader than the image/file OCR case in #15720.Reproduction Code
Run a
google.adkagent whose model (gemini-3.5-flash) returns a response whosecontent.partsincludes a part not handled byextract_message_from_part_google_genai(empty part, thought-signature part, orinline_data/file_datapart), with LLM Obs enabled. Thegoogle_adk.requestagent span is dropped and its childgoogle_genai.requestspans are orphaned.Error Logs
(Expect a companion
Error extracting LLMObs fields for span ... likely due to malformed dataimmediately before it.)Libraries in Use
Operating System
Linux (AWS ECS Fargate); also reproduces on Darwin 24.x
Related
Parthandling gap (google_utils.py:238); closednot_planned/stale but never fixed. This issue is thegoogle_adkconsequence: the unhandled part doesn't just render badly, it drops the whole agent span.