ci: remove dummy tracer #15715
ci: remove dummy tracer #15715
Conversation
|
|
mabdinur
force-pushed
the
munir/remove-dummy-tracer
branch
from
da6e2b9 to
040714e
Compare
mabdinur
force-pushed
the
munir/remove-dummy-tracer
branch
from
040714e to
2a9fa22
Compare
Performance SLOsComparing candidate munir/remove-dummy-tracer (0fc00b0) with baseline main (09ca1fb) ❌ Test Failures (1 suite)❌ iastaspects - 117/118✅ add_aspectTime: ✅ 17.982µs (SLO: <20.000µs 📉 -10.1%) vs baseline: 📈 +19.3% Memory: ✅ 42.762MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.8% ✅ add_inplace_aspectTime: ✅ 15.067µs (SLO: <20.000µs 📉 -24.7%) vs baseline: +0.7% Memory: ✅ 42.526MB (SLO: <43.250MB 🟡 -1.7%) vs baseline: +3.8% ✅ add_inplace_noaspectTime: ✅ 0.339µs (SLO: <10.000µs 📉 -96.6%) vs baseline: -0.4% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ add_noaspectTime: ✅ 0.548µs (SLO: <10.000µs 📉 -94.5%) vs baseline: ~same Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ bytearray_aspectTime: ✅ 18.053µs (SLO: <30.000µs 📉 -39.8%) vs baseline: +0.3% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.5% ✅ bytearray_extend_aspectTime: ✅ 24.074µs (SLO: <30.000µs 📉 -19.8%) vs baseline: +1.1% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +3.8% ✅ bytearray_extend_noaspectTime: ✅ 2.730µs (SLO: <10.000µs 📉 -72.7%) vs baseline: +3.2% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ bytearray_noaspectTime: ✅ 1.469µs (SLO: <10.000µs 📉 -85.3%) vs baseline: +1.1% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.2% ✅ bytes_aspectTime: ✅ 16.680µs (SLO: <20.000µs 📉 -16.6%) vs baseline: -0.9% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ bytes_noaspectTime: ✅ 1.416µs (SLO: <10.000µs 📉 -85.8%) vs baseline: +0.6% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ bytesio_aspectTime: ✅ 56.046µs (SLO: <70.000µs 📉 -19.9%) vs baseline: +1.3% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.3% ✅ bytesio_noaspectTime: ✅ 3.270µs (SLO: <10.000µs 📉 -67.3%) vs baseline: +2.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.6% ✅ capitalize_aspectTime: ✅ 14.766µs (SLO: <20.000µs 📉 -26.2%) vs baseline: +0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.2% ✅ capitalize_noaspectTime: ✅ 2.608µs (SLO: <10.000µs 📉 -73.9%) vs baseline: +0.9% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ casefold_aspectTime: ✅ 14.764µs (SLO: <20.000µs 📉 -26.2%) vs baseline: +0.4% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ casefold_noaspectTime: ✅ 3.153µs (SLO: <10.000µs 📉 -68.5%) vs baseline: +2.0% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ decode_aspectTime: ✅ 15.614µs (SLO: <30.000µs 📉 -48.0%) vs baseline: -0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.2% ✅ decode_noaspectTime: ✅ 1.613µs (SLO: <10.000µs 📉 -83.9%) vs baseline: +1.0% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.4% ✅ encode_aspectTime: ✅ 18.288µs (SLO: <30.000µs 📉 -39.0%) vs baseline: 📈 +23.0% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ encode_noaspectTime: ✅ 1.484µs (SLO: <10.000µs 📉 -85.2%) vs baseline: +0.4% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +5.4% ✅ format_aspectTime: ✅ 171.549µs (SLO: <200.000µs 📉 -14.2%) vs baseline: +0.6% Memory: ✅ 42.802MB (SLO: <43.250MB 🟡 -1.0%) vs baseline: +4.6% ✅ format_map_aspectTime: ✅ 191.396µs (SLO: <200.000µs -4.3%) vs baseline: -0.3% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.4% ✅ format_map_noaspectTime: ✅ 3.825µs (SLO: <10.000µs 📉 -61.8%) vs baseline: +2.2% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.2% ✅ format_noaspectTime: ✅ 3.153µs (SLO: <10.000µs 📉 -68.5%) vs baseline: +0.3% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.6% ✅ index_aspectTime: ✅ 15.331µs (SLO: <20.000µs 📉 -23.3%) vs baseline: +0.5% Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.6% ✅ index_noaspectTime: ✅ 0.461µs (SLO: <10.000µs 📉 -95.4%) vs baseline: -0.7% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ join_aspectTime: ✅ 17.092µs (SLO: <20.000µs 📉 -14.5%) vs baseline: +0.7% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.3% ✅ join_noaspectTime: ✅ 1.542µs (SLO: <10.000µs 📉 -84.6%) vs baseline: +2.9% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.1% ✅ ljust_aspectTime: ✅ 20.921µs (SLO: <30.000µs 📉 -30.3%) vs baseline: -0.2% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.1% ✅ ljust_noaspectTime: ✅ 2.679µs (SLO: <10.000µs 📉 -73.2%) vs baseline: +0.3% Memory: ✅ 42.743MB (SLO: <43.250MB 🟡 -1.2%) vs baseline: +5.1% ✅ lower_aspectTime: ✅ 17.915µs (SLO: <30.000µs 📉 -40.3%) vs baseline: +0.6% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.4% ✅ lower_noaspectTime: ✅ 2.438µs (SLO: <10.000µs 📉 -75.6%) vs baseline: +1.4% Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.8% ❌ lstrip_aspectTime: ❌ 21.881µs (SLO: <20.000µs +9.4%) vs baseline: 📈 +24.0% Memory: ✅ 42.723MB (SLO: <43.250MB 🟡 -1.2%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 1.876µs (SLO: <10.000µs 📉 -81.2%) vs baseline: +2.2% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ modulo_aspectTime: ✅ 166.053µs (SLO: <200.000µs 📉 -17.0%) vs baseline: ~same Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 174.599µs (SLO: <200.000µs 📉 -12.7%) vs baseline: ~same Memory: ✅ 43.195MB (SLO: <43.500MB 🟡 -0.7%) vs baseline: +5.5% ✅ modulo_aspect_for_bytesTime: ✅ 168.514µs (SLO: <200.000µs 📉 -15.7%) vs baseline: +0.4% Memory: ✅ 43.273MB (SLO: <43.500MB 🟡 -0.5%) vs baseline: +5.8% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.271µs (SLO: <200.000µs 📉 -13.9%) vs baseline: +0.4% Memory: ✅ 43.214MB (SLO: <43.500MB 🟡 -0.7%) vs baseline: +5.9% ✅ modulo_noaspectTime: ✅ 3.665µs (SLO: <10.000µs 📉 -63.4%) vs baseline: -1.6% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ replace_aspectTime: ✅ 212.244µs (SLO: <300.000µs 📉 -29.3%) vs baseline: -1.0% Memory: ✅ 43.018MB (SLO: <44.000MB -2.2%) vs baseline: +5.1% ✅ replace_noaspectTime: ✅ 2.904µs (SLO: <10.000µs 📉 -71.0%) vs baseline: 📉 -44.0% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ repr_aspectTime: ✅ 1.476µs (SLO: <10.000µs 📉 -85.2%) vs baseline: +7.7% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ repr_noaspectTime: ✅ 0.524µs (SLO: <10.000µs 📉 -94.8%) vs baseline: -1.6% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ rstrip_aspectTime: ✅ 19.061µs (SLO: <30.000µs 📉 -36.5%) vs baseline: ~same Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ rstrip_noaspectTime: ✅ 1.939µs (SLO: <10.000µs 📉 -80.6%) vs baseline: +2.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ slice_aspectTime: ✅ 15.940µs (SLO: <20.000µs 📉 -20.3%) vs baseline: -0.4% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.5% ✅ slice_noaspectTime: ✅ 0.598µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.2% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.3% ✅ stringio_aspectTime: ✅ 54.456µs (SLO: <80.000µs 📉 -31.9%) vs baseline: +1.1% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.9% ✅ stringio_noaspectTime: ✅ 3.656µs (SLO: <10.000µs 📉 -63.4%) vs baseline: +3.3% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ strip_aspectTime: ✅ 17.777µs (SLO: <20.000µs 📉 -11.1%) vs baseline: +0.5% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.5% ✅ strip_noaspectTime: ✅ 1.874µs (SLO: <10.000µs 📉 -81.3%) vs baseline: +2.2% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ swapcase_aspectTime: ✅ 18.615µs (SLO: <30.000µs 📉 -38.0%) vs baseline: +1.0% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.6% ✅ swapcase_noaspectTime: ✅ 2.806µs (SLO: <10.000µs 📉 -71.9%) vs baseline: +2.1% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ title_aspectTime: ✅ 18.433µs (SLO: <20.000µs -7.8%) vs baseline: +1.4% Memory: ✅ 42.644MB (SLO: <43.000MB 🟡 -0.8%) vs baseline: +4.4% ✅ title_noaspectTime: ✅ 2.685µs (SLO: <10.000µs 📉 -73.1%) vs baseline: +2.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ translate_aspectTime: ✅ 20.652µs (SLO: <30.000µs 📉 -31.2%) vs baseline: +1.1% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.0% ✅ translate_noaspectTime: ✅ 4.307µs (SLO: <10.000µs 📉 -56.9%) vs baseline: +1.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ upper_aspectTime: ✅ 18.126µs (SLO: <30.000µs 📉 -39.6%) vs baseline: +1.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.0% ✅ upper_noaspectTime: ✅ 2.451µs (SLO: <10.000µs 📉 -75.5%) vs baseline: +3.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.4% 📈 Performance Regressions (2 suites)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.150µs (SLO: <10.000µs 📉 -48.5%) vs baseline: 📈 +22.1% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 4.296µs (SLO: <10.000µs 📉 -57.0%) vs baseline: +0.1% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.7% ✅ ospathjoin_aspectTime: ✅ 6.268µs (SLO: <10.000µs 📉 -37.3%) vs baseline: +1.4% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathjoin_noaspectTime: ✅ 6.317µs (SLO: <10.000µs 📉 -36.8%) vs baseline: +1.5% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.7% ✅ ospathnormcase_aspectTime: ✅ 3.553µs (SLO: <10.000µs 📉 -64.5%) vs baseline: +0.7% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +5.0% ✅ ospathnormcase_noaspectTime: ✅ 3.640µs (SLO: <10.000µs 📉 -63.6%) vs baseline: +1.5% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.908µs (SLO: <10.000µs 📉 -50.9%) vs baseline: +1.6% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathsplit_noaspectTime: ✅ 5.029µs (SLO: <10.000µs 📉 -49.7%) vs baseline: +2.1% Memory: ✅ 41.504MB (SLO: <43.500MB -4.6%) vs baseline: +5.1% ✅ ospathsplitdrive_aspectTime: ✅ 3.774µs (SLO: <10.000µs 📉 -62.3%) vs baseline: +0.7% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathsplitdrive_noaspectTime: ✅ 0.748µs (SLO: <10.000µs 📉 -92.5%) vs baseline: +0.4% Memory: ✅ 41.327MB (SLO: <43.500MB -5.0%) vs baseline: +4.4% ✅ ospathsplitext_aspectTime: ✅ 4.647µs (SLO: <10.000µs 📉 -53.5%) vs baseline: -0.3% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 4.620µs (SLO: <10.000µs 📉 -53.8%) vs baseline: -1.2% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.393µs (SLO: <20.000µs 📉 -83.0%) vs baseline: 📈 +13.0% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.2% ✅ 1-count-metrics-100-timesTime: ✅ 200.428µs (SLO: <220.000µs -8.9%) vs baseline: +0.6% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.320µs (SLO: <20.000µs 📉 -83.4%) vs baseline: -1.5% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 1-distribution-metrics-100-timesTime: ✅ 215.046µs (SLO: <230.000µs -6.5%) vs baseline: -2.0% Memory: ✅ 34.937MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ 1-gauge-metric-1-timesTime: ✅ 2.207µs (SLO: <20.000µs 📉 -89.0%) vs baseline: -1.1% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ 1-gauge-metrics-100-timesTime: ✅ 137.647µs (SLO: <150.000µs -8.2%) vs baseline: +1.1% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-rate-metric-1-timesTime: ✅ 3.129µs (SLO: <20.000µs 📉 -84.4%) vs baseline: -0.2% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ 1-rate-metrics-100-timesTime: ✅ 212.759µs (SLO: <250.000µs 📉 -14.9%) vs baseline: -0.7% Memory: ✅ 35.016MB (SLO: <35.500MB 🟡 -1.4%) vs baseline: +5.3% ✅ 100-count-metrics-100-timesTime: ✅ 19.920ms (SLO: <22.000ms -9.5%) vs baseline: -2.5% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.6% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.258ms (SLO: <2.550ms 📉 -11.5%) vs baseline: -1.7% Memory: ✅ 35.212MB (SLO: <35.500MB 🟡 -0.8%) vs baseline: +5.0% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.400ms (SLO: <1.550ms -9.7%) vs baseline: +0.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 100-rate-metrics-100-timesTime: ✅ 2.192ms (SLO: <2.550ms 📉 -14.1%) vs baseline: -1.4% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ flush-1-metricTime: ✅ 4.547µs (SLO: <20.000µs 📉 -77.3%) vs baseline: -2.0% Memory: ✅ 35.114MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ flush-100-metricsTime: ✅ 173.940µs (SLO: <250.000µs 📉 -30.4%) vs baseline: -0.5% Memory: ✅ 35.291MB (SLO: <35.500MB 🟡 -0.6%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.175ms (SLO: <2.500ms 📉 -13.0%) vs baseline: ~same Memory: ✅ 35.940MB (SLO: <36.500MB 🟡 -1.5%) vs baseline: +4.7% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
Description
This PR standardizes how tests capture and verify trace data. Tests now use the global
ddtrace.tracerconfigured with aDummyWriterinstead of instantiatingDummyTracerdirectly. A newtest_spansfixture provides convenient access to captured spans viapop()andpop_traces()methods. The tracer state is properly reset between tests to prevent test pollution.Testing
Ran aiohttp, aiobotocore, asyncio, and subprocess test suites to verify the refactored fixtures work correctly.
Risks
Tests relying on tracer state leaking between tests may need adjustment.
Additional Notes
test_spansfixture and usetest_spans.pop()/test_spans.pop_traces()TracerTestCasesubclasses: useself.pop_spans()/self.pop_traces()