Nicole cybul/flaky test investigation #13253

ncybul · 2025-04-22T17:25:34Z

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2025-04-22T17:26:10Z

CODEOWNERS have been resolved as:

ddtrace/contrib/internal/litellm/utils.py                               @DataDog/apm-core-python @DataDog/apm-idm-python
releasenotes/notes/feat-llmobs-litellm-7fc73fd28ab88fac.yaml            @DataDog/apm-python
tests/contrib/litellm/cassettes/completion_multiple_choices_proxy.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_multiple_choices_with_tools.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_proxy.yaml                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_exclude_usage.yaml    @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_exclude_usage_proxy.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_exclude_usage_with_tools.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_multiple_choices_exclude_usage.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_multiple_choices_exclude_usage_proxy.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_multiple_choices_exclude_usage_with_tools.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_multiple_choices_proxy.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_multiple_choices_with_tools.yaml  @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_proxy.yaml            @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_stream_with_tools.yaml       @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/cassettes/completion_with_tools.yaml              @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/test_litellm_llmobs.py                            @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/litellm/patch.py                               @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/openai/utils.py                                @DataDog/ml-observability
ddtrace/llmobs/_integrations/litellm.py                                 @DataDog/ml-observability
ddtrace/llmobs/_integrations/openai.py                                  @DataDog/ml-observability
ddtrace/llmobs/_integrations/utils.py                                   @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
tests/contrib/litellm/conftest.py                                       @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/test_litellm.py                                   @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/litellm/utils.py                                          @DataDog/apm-core-python @DataDog/apm-idm-python

datadog-datadog-prod-us1 · 2025-04-22T17:26:10Z

ddtrace/llmobs/_integrations/utils.py

@@ -284,6 +289,187 @@ def get_messages_from_converse_content(role: str, content: list):
    return messages


+def openai_set_meta_tags_from_completion(span: Span, kwargs: Dict[str, Any], completions: Any) -> None:


🟠 Code Quality Violation

do not use Any, use a concrete type (...read more)

Use the Any type very carefully. Most of the time, the Any type is used because we do not know exactly what type is being used. If you want to specify that a value can be of any type, use object instead of Any.

Learn More

Python documentation: the Any type

datadog-datadog-prod-us1 · 2025-04-22T17:26:10Z

ddtrace/llmobs/_integrations/litellm.py

+        )
+
+    @staticmethod
+    def _extract_llmobs_metrics(resp: Any) -> Dict[str, Any]:


🟠 Code Quality Violation

do not use Any, use a concrete type (...read more)

Use the Any type very carefully. Most of the time, the Any type is used because we do not know exactly what type is being used. If you want to specify that a value can be of any type, use object instead of Any.

Learn More

Python documentation: the Any type

github-actions · 2025-04-22T17:46:33Z

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 234 ± 3 ms.

The average import time from base is: 250 ± 6 ms.

The import time difference between this PR and base is: -16.3 ± 0.2 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 13.548 ms (5.79%)

ddtrace 7.261 ms (3.10%)

ddtrace._logger 4.124 ms (1.76%)

ddtrace.internal.telemetry 3.706 ms (1.58%)

ddtrace.internal.telemetry.writer 3.602 ms (1.54%)

http.client 1.121 ms (0.48%)

email.parser 0.547 ms (0.23%)

email.feedparser 0.547 ms (0.23%)

email._policybase 0.547 ms (0.23%)

email.utils 0.419 ms (0.18%)

socket 0.159 ms (0.07%)

_socket 0.057 ms (0.02%)

selectors 0.035 ms (0.01%)

urllib.parse 0.110 ms (0.05%)

ipaddress 0.048 ms (0.02%)

email._parseaddr 0.107 ms (0.05%)

calendar 0.107 ms (0.05%)

locale 0.066 ms (0.03%)

random 0.020 ms (0.01%)

math 0.020 ms (0.01%)

email.header 0.109 ms (0.05%)

binascii 0.040 ms (0.02%)

email.base64mime 0.031 ms (0.01%)

base64 0.031 ms (0.01%)

struct 0.015 ms (0.01%)

_struct 0.015 ms (0.01%)

ssl 0.266 ms (0.11%)

_ssl 0.106 ms (0.05%)

email.message 0.031 ms (0.01%)

ddtrace.internal.utils.http 0.774 ms (0.33%)

dataclasses 0.398 ms (0.17%)

inspect 0.335 ms (0.14%)

dis 0.103 ms (0.04%)

opcode 0.051 ms (0.02%)

_opcode 0.027 ms (0.01%)

ast 0.092 ms (0.04%)

ddtrace.internal.http 0.222 ms (0.09%)

ddtrace.internal.runtime 0.174 ms (0.07%)

uuid 0.109 ms (0.05%)

platform 0.042 ms (0.02%)

_uuid 0.030 ms (0.01%)

ddtrace.internal.forksafe 0.066 ms (0.03%)

wrapt 0.038 ms (0.02%)

wrapt.__wrapt__ 0.038 ms (0.02%)

wrapt._wrappers 0.023 ms (0.01%)

ddtrace.internal.runtime.container 0.048 ms (0.02%)

ddtrace.internal.utils 0.033 ms (0.01%)

ddtrace.internal.utils.version 0.630 ms (0.27%)

ddtrace.vendor.packaging.version 0.630 ms (0.27%)

ddtrace.vendor.packaging 0.445 ms (0.19%)

ddtrace.vendor 0.445 ms (0.19%)

ddtrace.internal.module 0.445 ms (0.19%)

ddtrace.internal.wrapping.context 0.389 ms (0.17%)

ddtrace.internal.wrapping 0.365 ms (0.16%)

bytecode 0.310 ms (0.13%)

bytecode.bytecode 0.182 ms (0.08%)

bytecode.flags 0.154 ms (0.07%)

bytecode.instr 0.130 ms (0.06%)

bytecode.cfg 0.109 ms (0.05%)

bytecode.concrete 0.043 ms (0.02%)

ddtrace.internal.assembly 0.023 ms (0.01%)

contextvars 0.024 ms (0.01%)

_contextvars 0.024 ms (0.01%)

ddtrace.internal.encoding 0.330 ms (0.14%)

ddtrace.internal._encoding 0.060 ms (0.03%)

ddtrace.internal.telemetry.data 0.260 ms (0.11%)

ddtrace.internal.packages 0.205 ms (0.09%)

_sysconfigdata__linux_x86_64-linux-gnu 0.084 ms (0.04%)

ddtrace.settings.third_party 0.028 ms (0.01%)

sysconfig 0.055 ms (0.02%)

ddtrace.settings._agent 0.126 ms (0.05%)

ddtrace.settings._core 0.098 ms (0.04%)

ddtrace.internal.native 0.062 ms (0.03%)

ddtrace.internal.native._native 0.062 ms (0.03%)

ddtrace.settings._telemetry 0.100 ms (0.04%)

ddtrace.settings._inferred_base_service 0.060 ms (0.03%)

pathlib 0.060 ms (0.03%)

ddtrace.internal.telemetry.metrics_namespaces 0.085 ms (0.04%)

ddtrace.internal.periodic 0.060 ms (0.03%)

ddtrace.internal._threads 0.060 ms (0.03%)

ddtrace.internal.atexit 0.027 ms (0.01%)

signal 0.027 ms (0.01%)

ddtrace.internal.telemetry.logging 0.026 ms (0.01%)

ddtrace.internal.telemetry.constants 0.029 ms (0.01%)

logging 0.323 ms (0.14%)

traceback 0.257 ms (0.11%)

contextlib 0.257 ms (0.11%)

typing 0.095 ms (0.04%)

ddtrace.trace 1.014 ms (0.43%)

ddtrace._trace.filters 0.595 ms (0.25%)

ddtrace._trace.processor 0.595 ms (0.25%)

ddtrace._trace.sampler 0.323 ms (0.14%)

ddtrace._trace.span 0.289 ms (0.12%)

ddtrace.internal.sampling 0.100 ms (0.04%)

ddtrace.internal.rate_limiter 0.048 ms (0.02%)

ddtrace.internal._rand 0.058 ms (0.02%)

pprint 0.040 ms (0.02%)

ddtrace._trace._span_pointer 0.040 ms (0.02%)

ddtrace.internal.dogstatsd 0.143 ms (0.06%)

ddtrace.vendor.dogstatsd 0.143 ms (0.06%)

ddtrace.vendor.dogstatsd.base 0.143 ms (0.06%)

ddtrace.vendor.dogstatsd.container 0.027 ms (0.01%)

queue 0.025 ms (0.01%)

heapq 0.025 ms (0.01%)

_heapq 0.025 ms (0.01%)

ddtrace.internal.writer 0.076 ms (0.03%)

ddtrace.internal.writer.writer 0.076 ms (0.03%)

ddtrace._trace.tracer 0.187 ms (0.08%)

ddtrace.internal.processor.endpoint_call_counter 0.066 ms (0.03%)

ddtrace.internal.debug 0.028 ms (0.01%)

ddtrace.internal.peer_service.processor 0.021 ms (0.01%)

ddtrace._trace.context 0.103 ms (0.04%)

ddtrace._trace._span_link 0.051 ms (0.02%)

ddtrace.settings._config 0.960 ms (0.41%)

ddtrace.internal.gitmetadata 0.387 ms (0.17%)

ddtrace.ext.ci 0.368 ms (0.16%)

ddtrace.ext.git 0.316 ms (0.14%)

shutil 0.128 ms (0.05%)

zlib 0.032 ms (0.01%)

lzma 0.022 ms (0.01%)

subprocess 0.085 ms (0.04%)

fcntl 0.017 ms (0.01%)

tempfile 0.034 ms (0.01%)

ddtrace.internal._file_queue 0.081 ms (0.03%)

secrets 0.081 ms (0.03%)

hmac 0.081 ms (0.03%)

_hashlib 0.057 ms (0.02%)

ddtrace.internal.schema 0.034 ms (0.01%)

ddtrace.internal.schema.span_attribute_schema 0.034 ms (0.01%)

ddtrace.settings.endpoint_config 0.020 ms (0.01%)

ddtrace._monkey 0.306 ms (0.13%)

ddtrace.appsec._listeners 0.163 ms (0.07%)

ddtrace.internal.core 0.163 ms (0.07%)

ddtrace.internal.core.event_hub 0.105 ms (0.05%)

ddtrace.settings.asm 0.067 ms (0.03%)

ddtrace.internal._unpatched 0.135 ms (0.06%)

json 0.089 ms (0.04%)

json.decoder 0.089 ms (0.04%)

re 0.060 ms (0.03%)

re._compiler 0.031 ms (0.01%)

enum 0.030 ms (0.01%)

functools 0.030 ms (0.01%)

json.scanner 0.029 ms (0.01%)

_json 0.029 ms (0.01%)

threading 0.046 ms (0.02%)

ddtrace.bootstrap.sitecustomize 6.288 ms (2.69%)

ddtrace.bootstrap.preload 5.543 ms (2.37%)

ddtrace.internal.products 3.822 ms (1.63%)

ddtrace.internal.remoteconfig.client 0.960 ms (0.41%)

importlib.metadata 0.640 ms (0.27%)

importlib.abc 0.342 ms (0.15%)

importlib.resources 0.316 ms (0.13%)

importlib.resources._common 0.316 ms (0.13%)

zipfile 0.082 ms (0.04%)

csv 0.077 ms (0.03%)

_csv 0.043 ms (0.02%)

importlib.metadata._meta 0.031 ms (0.01%)

ddtrace.internal.symbol_db.remoteconfig 0.339 ms (0.14%)

ddtrace.internal.symbol_db.symbols 0.236 ms (0.10%)

gzip 0.065 ms (0.03%)

ddtrace.settings.symbol_db 0.036 ms (0.02%)

multiprocessing 0.216 ms (0.09%)

multiprocessing.context 0.216 ms (0.09%)

multiprocessing.reduction 0.186 ms (0.08%)

pickle 0.186 ms (0.08%)

_pickle 0.043 ms (0.02%)

_compat_pickle 0.030 ms (0.01%)

multiprocessing.process 0.030 ms (0.01%)

ddtrace.internal.remoteconfig._connectors 0.110 ms (0.05%)

ctypes 0.073 ms (0.03%)

_ctypes 0.073 ms (0.03%)

multiprocessing.sharedctypes 0.091 ms (0.04%)

multiprocessing.heap 0.057 ms (0.02%)

multiprocessing.util 0.031 ms (0.01%)

mmap 0.025 ms (0.01%)

ddtrace.internal.remoteconfig.worker 0.071 ms (0.03%)

ddtrace.settings.dynamic_instrumentation 0.047 ms (0.02%)

ddtrace.settings.profiling 1.006 ms (0.43%)

ddtrace.vendor.psutil 0.851 ms (0.36%)

ddtrace.vendor.psutil._common 0.445 ms (0.19%)

ddtrace.vendor.psutil._pslinux 0.238 ms (0.10%)

glob 0.039 ms (0.02%)

ddtrace.vendor.psutil._psutil_linux 0.037 ms (0.02%)

ddtrace.vendor.psutil._compat 0.044 ms (0.02%)

ddtrace.internal.datadog.profiling.ddup 0.058 ms (0.02%)

ddtrace.internal.datadog.profiling.ddup._ddup 0.058 ms (0.02%)

ddtrace.internal.flare.flare 0.186 ms (0.08%)

logging.handlers 0.086 ms (0.04%)

ddtrace.appsec._remoteconfiguration 0.115 ms (0.05%)

ddtrace.appsec._capabilities 0.053 ms (0.02%)

ddtrace.settings.crashtracker 0.081 ms (0.03%)

ddtrace.internal.datadog.profiling.crashtracker 0.081 ms (0.03%)

ddtrace.internal.datadog.profiling.crashtracker._crashtracker 0.081 ms (0.03%)

ddtrace.internal.runtime.runtime_metrics 0.043 ms (0.02%)

ddtrace.internal.core.crashtracking 0.029 ms (0.01%)

ddtrace.internal.flare.handler 0.024 ms (0.01%)

ddtrace._trace.trace_handlers 0.339 ms (0.14%)

ddtrace._trace._inferred_proxy 0.165 ms (0.07%)

ddtrace.propagation.http 0.141 ms (0.06%)

ddtrace.internal._tagset 0.051 ms (0.02%)

ddtrace.contrib.trace_utils 0.089 ms (0.04%)

ddtrace.contrib.internal.trace_utils 0.089 ms (0.04%)

ddtrace.contrib.internal.trace_utils_base 0.038 ms (0.02%)

ddtrace.appsec._common_module_patches 0.144 ms (0.06%)

ddtrace.appsec._asm_request_context 0.087 ms (0.04%)

ddtrace.appsec._utils 0.033 ms (0.01%)

ddtrace.appsec._metrics 0.023 ms (0.01%)

shlex 0.033 ms (0.01%)

pr-commenter · 2025-04-22T18:45:27Z

Benchmarks

Benchmark execution time: 2025-04-24 13:46:04

Comparing candidate commit 31209d5 in PR branch nicole-cybul/flaky-test-investigation with baseline commit 1871f28 in branch main.

Found 5 performance improvements and 2 performance regressions! Performance is the same for 479 metrics, 2 unstable metrics.

scenario:ddtracerun-auto_profiling

🟥 execution_time [+171.113ms; +174.022ms] or [+26.137%; +26.582%]

scenario:otelspan-start-finish

🟩 execution_time [-18.150ms; -17.750ms] or [-23.011%; -22.503%]

scenario:otelspan-start-finish-telemetry

🟩 execution_time [-19.683ms; -19.228ms] or [-24.405%; -23.840%]

scenario:span-start-finish

🟩 execution_time [-16.145ms; -15.533ms] or [-33.427%; -32.160%]

scenario:span-start-finish-telemetry

🟩 execution_time [-16.315ms; -15.734ms] or [-33.043%; -31.866%]

scenario:span-start-finish-traceid128

🟩 execution_time [-18.532ms; -17.998ms] or [-36.099%; -35.058%]

scenario:telemetryaddmetric-flush-1-metric

🟥 execution_time [+410.428ns; +555.740ns] or [+9.448%; +12.793%]

ncybul added 30 commits March 19, 2025 14:03

add litellm apm integration

69d9709

add tests for litellm

f6c9ead

reuse cassettes that are similar across different litellm requests

7e43133

add test snapshots

65714e5

trace get_llm_provider

462b576

remove provider tagging for now

a01b415

test out other models

4869794

add global tags test

0adc8e4

add release note

800fcf4

run black

1cc21e1

fix requirements lock file

b95d96f

remove unnecessary bedrock credentials in tests

9f63cee

add documentation

a066c99

add llmobs base span tag method for litellm

010f85e

add provider tagging and stream capturing support

528a0ec

attach usage metrics to llmobs spans

4b16a4c

finish non streamed spans

0a6b744

use sample pc config

33aae75

move openai message parsing utils to shared utils file

184d728

reuse role for litellm streamed multi choice responses

a241160

pass operation to litellm llmobs set tags

2d2337a

wrap get_llm_provider in litellm sdk

3891da9

update provider map to store parsed model name and provider

df8449e

update model name based on model map

122cf45

add llmobs test file and remove usage extraction logic

92670c5

add more tests

60830c4

run black

f4d76d7

add tool call tests

a4a2b24

add test for case where integrations are enabled

64f48a1

add tests for proxy requests

4dd467c

ncybul added 17 commits April 1, 2025 10:03

add litellm tests to suitespec

f0b8a72

Merge branch 'main' into nicole-cybul/litellm-llmobs-tracing

2ed9abb

consume streams for apm tests

8d5d24b

remove unnecessary snapshot files and riot requirements

377b40d

fix merge conflict bug for openai span linking

d4185d8

fix integrations enabled test

48e57c3

add next and anext methods to traced stream classes

252f092

remove unnecessary config stuff

07685a8

add release note

c273b52

fix errored out streamed requests not setting llmobs tags properly

5ee0f03

run black

a5cdfa5

Merge branch 'main' into nicole-cybul/litellm-llmobs-tracing

c92cc9d

style fixes

1474258

use wrapt.ObjectProxy for traced streams

c9ba90a

small improvements to model name and metric extraction

b2e8f59

cleaned up configuration for litellm tests

4b14921

style fixes

73417db

datadog-datadog-prod-us1 bot reviewed Apr 22, 2025

View reviewed changes

removed some tests

2f9da4e

ncybul added 2 commits April 24, 2025 13:40

remove include usage param

50b5e4c

add exclude usage test

31209d5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nicole cybul/flaky test investigation #13253

Nicole cybul/flaky test investigation #13253

ncybul commented Apr 22, 2025

github-actions bot commented Apr 22, 2025

datadog-datadog-prod-us1 bot Apr 22, 2025

datadog-datadog-prod-us1 bot Apr 22, 2025

github-actions bot commented Apr 22, 2025 •

edited

Loading

pr-commenter bot commented Apr 22, 2025 •

edited

Loading

		@@ -284,6 +289,187 @@ def get_messages_from_converse_content(role: str, content: list):
		return messages


		def openai_set_meta_tags_from_completion(span: Span, kwargs: Dict[str, Any], completions: Any) -> None:

Nicole cybul/flaky test investigation #13253

Are you sure you want to change the base?

Nicole cybul/flaky test investigation #13253

Conversation

ncybul commented Apr 22, 2025

Checklist

Reviewer Checklist

github-actions bot commented Apr 22, 2025

datadog-datadog-prod-us1 bot Apr 22, 2025

Choose a reason for hiding this comment

🟠 Code Quality Violation

Learn More

datadog-datadog-prod-us1 bot Apr 22, 2025

Choose a reason for hiding this comment

🟠 Code Quality Violation

Learn More

github-actions bot commented Apr 22, 2025 • edited Loading

Bootstrap import analysis

Summary

Import time breakdown

pr-commenter bot commented Apr 22, 2025 • edited Loading

Benchmarks

scenario:ddtracerun-auto_profiling

scenario:otelspan-start-finish

scenario:otelspan-start-finish-telemetry

scenario:span-start-finish

scenario:span-start-finish-telemetry

scenario:span-start-finish-traceid128

scenario:telemetryaddmetric-flush-1-metric

github-actions bot commented Apr 22, 2025 •

edited

Loading

pr-commenter bot commented Apr 22, 2025 •

edited

Loading