-
Notifications
You must be signed in to change notification settings - Fork 474
fix(langchain): correctly identify Google GenAI models #15707
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
Performance SLOsComparing candidate alex/MLOS-338_fix-langchain-google-genai (b85a364) with baseline main (6653a0e) ❌ Test Failures (1 suite)❌ iastaspects - 117/118✅ add_aspectTime: ✅ 18.003µs (SLO: <20.000µs -10.0%) vs baseline: 📈 +19.5% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.1% ✅ add_inplace_aspectTime: ✅ 14.919µs (SLO: <20.000µs 📉 -25.4%) vs baseline: -0.8% Memory: ✅ 42.782MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.4% ✅ add_inplace_noaspectTime: ✅ 0.341µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +0.9% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.5% ✅ add_noaspectTime: ✅ 0.545µs (SLO: <10.000µs 📉 -94.5%) vs baseline: -0.1% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ bytearray_aspectTime: ✅ 18.057µs (SLO: <30.000µs 📉 -39.8%) vs baseline: ~same Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.5% ✅ bytearray_extend_aspectTime: ✅ 23.882µs (SLO: <30.000µs 📉 -20.4%) vs baseline: ~same Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ bytearray_extend_noaspectTime: ✅ 2.738µs (SLO: <10.000µs 📉 -72.6%) vs baseline: -0.4% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 1.460µs (SLO: <10.000µs 📉 -85.4%) vs baseline: -1.0% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.1% ✅ bytes_aspectTime: ✅ 16.692µs (SLO: <20.000µs 📉 -16.5%) vs baseline: +0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.0% ✅ bytes_noaspectTime: ✅ 1.428µs (SLO: <10.000µs 📉 -85.7%) vs baseline: +1.2% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.2% ✅ bytesio_aspectTime: ✅ 55.818µs (SLO: <70.000µs 📉 -20.3%) vs baseline: -0.2% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.3% ✅ bytesio_noaspectTime: ✅ 3.271µs (SLO: <10.000µs 📉 -67.3%) vs baseline: -0.3% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.3% ✅ capitalize_aspectTime: ✅ 14.705µs (SLO: <20.000µs 📉 -26.5%) vs baseline: +0.5% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.5% ✅ capitalize_noaspectTime: ✅ 2.597µs (SLO: <10.000µs 📉 -74.0%) vs baseline: -0.2% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ casefold_aspectTime: ✅ 14.727µs (SLO: <20.000µs 📉 -26.4%) vs baseline: +0.5% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.4% ✅ casefold_noaspectTime: ✅ 3.141µs (SLO: <10.000µs 📉 -68.6%) vs baseline: -0.7% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ decode_aspectTime: ✅ 15.577µs (SLO: <30.000µs 📉 -48.1%) vs baseline: -0.7% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.5% ✅ decode_noaspectTime: ✅ 1.619µs (SLO: <10.000µs 📉 -83.8%) vs baseline: +0.8% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.3% ✅ encode_aspectTime: ✅ 18.290µs (SLO: <30.000µs 📉 -39.0%) vs baseline: 📈 +23.1% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ encode_noaspectTime: ✅ 1.495µs (SLO: <10.000µs 📉 -85.1%) vs baseline: +0.4% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ format_aspectTime: ✅ 170.978µs (SLO: <200.000µs 📉 -14.5%) vs baseline: -0.3% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.5% ✅ format_map_aspectTime: ✅ 191.107µs (SLO: <200.000µs -4.4%) vs baseline: ~same Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.2% ✅ format_map_noaspectTime: ✅ 3.771µs (SLO: <10.000µs 📉 -62.3%) vs baseline: -1.3% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.7% ✅ format_noaspectTime: ✅ 3.163µs (SLO: <10.000µs 📉 -68.4%) vs baseline: +1.2% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 15.315µs (SLO: <20.000µs 📉 -23.4%) vs baseline: -0.4% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +4.2% ✅ index_noaspectTime: ✅ 0.461µs (SLO: <10.000µs 📉 -95.4%) vs baseline: -0.3% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.6% ✅ join_aspectTime: ✅ 17.145µs (SLO: <20.000µs 📉 -14.3%) vs baseline: ~same Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.3% ✅ join_noaspectTime: ✅ 1.561µs (SLO: <10.000µs 📉 -84.4%) vs baseline: +0.6% Memory: ✅ 42.762MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +5.2% ✅ ljust_aspectTime: ✅ 20.800µs (SLO: <30.000µs 📉 -30.7%) vs baseline: -0.2% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +3.9% ✅ ljust_noaspectTime: ✅ 2.698µs (SLO: <10.000µs 📉 -73.0%) vs baseline: +0.4% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.5% ✅ lower_aspectTime: ✅ 17.939µs (SLO: <30.000µs 📉 -40.2%) vs baseline: -0.4% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ lower_noaspectTime: ✅ 2.444µs (SLO: <10.000µs 📉 -75.6%) vs baseline: +0.4% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.7% ✅ lstrip_aspectTime: ✅ 17.797µs (SLO: <20.000µs 📉 -11.0%) vs baseline: +0.7% Memory: ✅ 42.939MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.9% ✅ lstrip_noaspectTime: ✅ 1.873µs (SLO: <10.000µs 📉 -81.3%) vs baseline: ~same Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.5% ✅ modulo_aspectTime: ✅ 165.949µs (SLO: <200.000µs 📉 -17.0%) vs baseline: -0.3% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.5% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 175.025µs (SLO: <200.000µs 📉 -12.5%) vs baseline: +0.5% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ modulo_aspect_for_bytesTime: ✅ 168.541µs (SLO: <200.000µs 📉 -15.7%) vs baseline: +0.1% Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.4% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.192µs (SLO: <200.000µs 📉 -13.9%) vs baseline: ~same Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.2% ✅ modulo_noaspectTime: ✅ 3.746µs (SLO: <10.000µs 📉 -62.5%) vs baseline: +1.4% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ replace_aspectTime: ✅ 212.840µs (SLO: <300.000µs 📉 -29.1%) vs baseline: +0.5% Memory: ✅ 43.136MB (SLO: <44.000MB 🟡 -2.0%) vs baseline: +4.8% ✅ replace_noaspectTime: ✅ 2.901µs (SLO: <10.000µs 📉 -71.0%) vs baseline: +0.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ repr_aspectTime: ✅ 1.406µs (SLO: <10.000µs 📉 -85.9%) vs baseline: -1.1% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ repr_noaspectTime: ✅ 0.527µs (SLO: <10.000µs 📉 -94.7%) vs baseline: +0.5% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ rstrip_aspectTime: ✅ 19.124µs (SLO: <30.000µs 📉 -36.3%) vs baseline: +0.5% Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.5% ✅ rstrip_noaspectTime: ✅ 2.036µs (SLO: <10.000µs 📉 -79.6%) vs baseline: +6.0% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ slice_aspectTime: ✅ 15.979µs (SLO: <20.000µs 📉 -20.1%) vs baseline: +0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ slice_noaspectTime: ✅ 0.605µs (SLO: <10.000µs 📉 -93.9%) vs baseline: +0.9% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ stringio_aspectTime: ✅ 54.360µs (SLO: <80.000µs 📉 -32.1%) vs baseline: +0.1% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ stringio_noaspectTime: ✅ 3.649µs (SLO: <10.000µs 📉 -63.5%) vs baseline: +0.1% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.6% ✅ strip_aspectTime: ✅ 17.621µs (SLO: <20.000µs 📉 -11.9%) vs baseline: -0.1% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.4% ✅ strip_noaspectTime: ✅ 1.860µs (SLO: <10.000µs 📉 -81.4%) vs baseline: -0.3% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.5% ✅ swapcase_aspectTime: ✅ 18.562µs (SLO: <30.000µs 📉 -38.1%) vs baseline: -0.1% Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.7% ✅ swapcase_noaspectTime: ✅ 2.812µs (SLO: <10.000µs 📉 -71.9%) vs baseline: -0.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.5% ❌ title_aspectTime: ❌ 22.060µs (SLO: <20.000µs 📈 +10.3%) vs baseline: 📈 +20.5% Memory: ✅ 42.723MB (SLO: <43.000MB 🟡 -0.6%) vs baseline: +4.3% ✅ title_noaspectTime: ✅ 2.690µs (SLO: <10.000µs 📉 -73.1%) vs baseline: -1.2% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.3% ✅ translate_aspectTime: ✅ 20.551µs (SLO: <30.000µs 📉 -31.5%) vs baseline: -0.5% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.2% ✅ translate_noaspectTime: ✅ 4.352µs (SLO: <10.000µs 📉 -56.5%) vs baseline: +0.2% Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +5.3% ✅ upper_aspectTime: ✅ 17.968µs (SLO: <30.000µs 📉 -40.1%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +3.8% ✅ upper_noaspectTime: ✅ 2.442µs (SLO: <10.000µs 📉 -75.6%) vs baseline: -0.7% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.2% 📈 Performance Regressions (2 suites)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.116µs (SLO: <10.000µs 📉 -48.8%) vs baseline: 📈 +20.3% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 4.300µs (SLO: <10.000µs 📉 -57.0%) vs baseline: +0.5% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathjoin_aspectTime: ✅ 6.218µs (SLO: <10.000µs 📉 -37.8%) vs baseline: -0.7% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathjoin_noaspectTime: ✅ 6.335µs (SLO: <10.000µs 📉 -36.7%) vs baseline: +0.7% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +5.0% ✅ ospathnormcase_aspectTime: ✅ 3.568µs (SLO: <10.000µs 📉 -64.3%) vs baseline: -0.4% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.6% ✅ ospathnormcase_noaspectTime: ✅ 3.615µs (SLO: <10.000µs 📉 -63.9%) vs baseline: +0.1% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.912µs (SLO: <10.000µs 📉 -50.9%) vs baseline: +0.1% Memory: ✅ 41.347MB (SLO: <43.500MB -5.0%) vs baseline: +4.6% ✅ ospathsplit_noaspectTime: ✅ 5.021µs (SLO: <10.000µs 📉 -49.8%) vs baseline: ~same Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.6% ✅ ospathsplitdrive_aspectTime: ✅ 3.765µs (SLO: <10.000µs 📉 -62.3%) vs baseline: +0.3% Memory: ✅ 41.347MB (SLO: <43.500MB -5.0%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.751µs (SLO: <10.000µs 📉 -92.5%) vs baseline: -0.2% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathsplitext_aspectTime: ✅ 4.631µs (SLO: <10.000µs 📉 -53.7%) vs baseline: +1.0% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 4.611µs (SLO: <10.000µs 📉 -53.9%) vs baseline: -1.3% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.9% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.371µs (SLO: <20.000µs 📉 -83.1%) vs baseline: 📈 +14.1% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-count-metrics-100-timesTime: ✅ 199.606µs (SLO: <220.000µs -9.3%) vs baseline: +0.7% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-distribution-metric-1-timesTime: ✅ 3.302µs (SLO: <20.000µs 📉 -83.5%) vs baseline: -0.3% Memory: ✅ 34.937MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.0% ✅ 1-distribution-metrics-100-timesTime: ✅ 212.852µs (SLO: <230.000µs -7.5%) vs baseline: -0.3% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ 1-gauge-metric-1-timesTime: ✅ 2.187µs (SLO: <20.000µs 📉 -89.1%) vs baseline: -0.3% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.595µs (SLO: <150.000µs -8.9%) vs baseline: -0.2% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 1-rate-metric-1-timesTime: ✅ 3.112µs (SLO: <20.000µs 📉 -84.4%) vs baseline: -0.6% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 1-rate-metrics-100-timesTime: ✅ 211.503µs (SLO: <250.000µs 📉 -15.4%) vs baseline: -0.4% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.2% ✅ 100-count-metrics-100-timesTime: ✅ 20.048ms (SLO: <22.000ms -8.9%) vs baseline: +0.3% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.251ms (SLO: <2.550ms 📉 -11.7%) vs baseline: +1.8% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.408ms (SLO: <1.550ms -9.1%) vs baseline: -0.1% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ 100-rate-metrics-100-timesTime: ✅ 2.214ms (SLO: <2.550ms 📉 -13.2%) vs baseline: +0.5% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ flush-1-metricTime: ✅ 4.561µs (SLO: <20.000µs 📉 -77.2%) vs baseline: +0.5% Memory: ✅ 35.134MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +4.7% ✅ flush-100-metricsTime: ✅ 173.114µs (SLO: <250.000µs 📉 -30.8%) vs baseline: ~same Memory: ✅ 35.173MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.5% ✅ flush-1000-metricsTime: ✅ 2.168ms (SLO: <2.500ms 📉 -13.3%) vs baseline: -0.3% Memory: ✅ 36.078MB (SLO: <36.500MB 🟡 -1.2%) vs baseline: +4.8% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
Description
Fixes incorrect
model_providertagging for LangChain Google Generative AI (Gemini) LLM spans.When using
ChatGoogleGenerativeAIfromlangchain_google_genai, the spans were tagged withmodel_provider: "chat"instead ofmodel_provider: "google", causing cost estimation to fail (showing "N/A") in LLM Observability.Root cause: The provider was extracted directly from
instance._llm_type, butChatGoogleGenerativeAI._llm_typereturns"chat"(a generic LangChain convention), not a Google-specific identifier.Solution: Added
_extract_provider()helper function that:"google")instance._llm_typefor other providersAlso:
"models/gemini-2.0-flash"→"gemini-2.0-flash") for consistency with direct Google GenAI integrationddtrace/llmobs/_integrations/model_providers.pyfor reuse across integrations (LangChain, Google GenAI, etc.)Testing
test_llmobs_google_genai_chat_model- verifies provider detection for basic Gemini modeltest_llmobs_google_genai_chat_model_with_path- verifies model name normalization with path prefixRisks
🤷
Additional Notes
The versions used in the riotfile's venvs are a bit arbitrary. It would seem to be that the versioning is more "major-version-oriented" and "minor-version-oriented" unlike the other packages. I guess latest is the most important anyways.