Skip to content

[BUG]: google_adk LLMObs drops agent span ("missing span kind") when a Gemini response Part is unhandled #18698

Description

@dpolistwm

Tracer Version(s)

4.10.5 (also reproduces on main)

Python Version(s)

3.12.13

Pip Version(s)

uv 0.x / pip 24.x

Bug Report

The google_adk LLM Observability integration intermittently drops the agent/request span entirely, emitting this error and leaving the child google_genai.request LLM spans orphaned (their parent_id points at a span that no longer exists in LLM Obs):

ERROR ddtrace.llmobs._llmobs: Error preparing LLMObs span event for span
<Span(id=...,name=google_adk.request)>, missing span kind in span context.

Root cause is an ordering bug in GoogleAdkIntegration._llmobs_set_tags. The span-kind annotation is the last statement, and it runs only if the preceding operation-specific extractor returns normally:

https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_integrations/google_adk.py#L30-L50

def _llmobs_set_tags(self, span, args, kwargs, response=None, operation=""):
    if operation == "agent":
        self._llmobs_set_tags_agent(span, args, kwargs, response)   # <-- may raise
    elif operation == "tool":
        self._llmobs_set_tags_tool(span, args, kwargs, response)
    elif operation == "code_execute":
        self._llmobs_set_tags_code_execute(span, args, kwargs, response)

    _annotate_llmobs_span_data(                                     # <-- ONLY place kind is set
        span,
        kind=operation,
        model_name=span.get_tag("google_adk.request.model") or "",
        model_provider=span.get_tag("google_adk.request.provider") or "",
    )

The caller swallows-and-logs any exception from _llmobs_set_tags:

https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_integrations/base.py#L114-L119

try:
    self._llmobs_set_tags(span, args, kwargs, response, operation)
except Exception:
    log.error("Error extracting LLMObs fields for span %s, likely due to malformed data", span, ...)

So if _llmobs_set_tags_agent (or _tag_agent_manifest) raises, the _annotate_llmobs_span_data(kind=...) line never executes → the span finishes with no kind → _prepare_llmobs_span_data rejects it and the span is dropped:

https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_llmobs.py#L614-L619

Two independent defects feed this:

  1. Fragile orderingkind=operation should be committed before the fallible operation-specific extraction (or in a finally), so a failure in input/output/manifest extraction degrades gracefully (empty I/O) instead of dropping the whole span and orphaning its children.

  2. The thing that raises is the same Gemini Part handling gap tracked in [BUG]: LLM Obs does not correctly handle google FilePart / ImagePart type #15720 ("LLM Obs does not correctly handle google FilePart / ImagePart type", closed not_planned by the stale bot, still unfixed on main). extract_message_from_part_google_genai has no branch for inline_data / file_data / empty / thought-signature parts and falls through to:

https://github.com/DataDog/dd-trace-py/blob/v4.10.5/ddtrace/llmobs/_integrations/google_utils.py#L238

In the google_genai LLM-span path that fallback merely produces a confusing Unsupported file type: <class 'google.genai.types.Part'> string in the output (the #15720 symptom). In the google_adk agent-span path, the same malformed parts can take down the entire _llmobs_set_tags_agent extraction, which is what triggers the span drop described here.

We observe this with gemini-3.5-flash returning a normal text answer plus an extra empty / thought-signature Part — no images involved — so the trigger is broader than the image/file OCR case in #15720.

Reproduction Code

Run a google.adk agent whose model (gemini-3.5-flash) returns a response whose content.parts includes a part not handled by extract_message_from_part_google_genai (empty part, thought-signature part, or inline_data/file_data part), with LLM Obs enabled. The google_adk.request agent span is dropped and its child google_genai.request spans are orphaned.

Error Logs

Error preparing LLMObs span event for span <Span(id=14479202674267337881,name=google_adk.request)>, missing span kind in span context.

(Expect a companion Error extracting LLMObs fields for span ... likely due to malformed data immediately before it.)

Libraries in Use

  • google-adk 2.2.0
  • google-genai 2.8.0

Operating System

Linux (AWS ECS Fargate); also reproduces on Darwin 24.x

Related

Metadata

Metadata

Assignees

Labels

LLMObsLLM Observability related

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions