Conversation
Same pattern as DI.exception_message: reads the internal `bt` ivar directly via rb_ivar_get, bypassing any Ruby-level override of Exception#backtrace. This ensures DI instrumentation never invokes customer code when serializing exception data. - Added exception_backtrace to ext/libdatadog_api/di.c - Updated serialize_throwable to use DI.exception_backtrace - Added RBS signature - Added unit tests for the C extension method - Added integration test for backtrace override bypass Co-Authored-By: Claude <noreply@anthropic.com>
When CodeTracker starts, use the all_iseqs C extension to populate the registry with instruction sequences for files that were loaded before tracking began. This enables line probes on third-party code and application code loaded at boot time. Only whole-file iseqs (first_lineno == 0) are backfilled — per-method iseqs require instrumenter changes to select the correct iseq for a target line and will be supported in a follow-up. Backfill does not overwrite entries from :script_compiled, which are authoritative. The C extension availability is checked via DI.respond_to?(:all_iseqs) so the code gracefully degrades when the extension is not compiled. - Added CodeTracker#backfill_registry - Called from CodeTracker#start after trace point is enabled - Added RBS signature - Added tests for backfill behavior and C extension fallback Co-Authored-By: Claude <noreply@anthropic.com>
- Added rescue block around backfill_registry so failures are best-effort (logged + telemetry) rather than propagating - Replaced all skip-based tests with mock-based tests that exercise backfill logic without requiring the compiled C extension - Added tests for: mixed iseq types, multiple files, error handling, suffix/exact lookup on backfilled entries, start ordering - 27 examples, 0 failures, 0 pending, 0 skipped Co-Authored-By: Claude <noreply@anthropic.com>
Tests the end-to-end flow: test class loaded before code tracking starts → CodeTracker#start triggers backfill via all_iseqs C extension → iseq recovered from object space → line probe installed on backfilled iseq → probe fires and captures local variables. Runs under rake spec:di_with_ext (requires compiled C extension). Three test cases: - Probe installs successfully on backfilled iseq - Probe fires when target line executes - Snapshot captures local variables from backfilled iseq Co-Authored-By: Claude <noreply@anthropic.com>
On macOS CI the C extension is compiled, so backfill_registry populates the CodeTracker registry with pre-loaded files during start. This broke existing tests that expect the registry to be empty after start or to contain exactly N explicitly-loaded files. Fix by stubbing backfill_registry in test contexts that exercise :script_compiled behavior. Backfill is tested separately in its own describe blocks. Affected contexts: - CodeTracker #start (before block) - CodeTracker shared context 'when code tracker is running' - CodeTracker #iseqs_for_path_suffix (around block) - Instrumenter shared context 'with code tracking' Co-Authored-By: Claude <noreply@anthropic.com>
…kfill The backfill filter used first_lineno == 0 to identify whole-file iseqs, but most whole-file iseqs from all_iseqs have first_lineno == 1. The new DI.iseq_type method reads the iseq type directly from the Ruby VM struct and returns a symbol (:top, :method, :block, :class, etc.). The backfill now filters by type == :top || type == :main, which correctly identifies whole-file iseqs regardless of first_lineno. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
rb_iseq_type is an internal Ruby function that only exists in Ruby 3.1+. On Ruby 2.7 and 3.0, referencing it causes an undefined symbol error at load time, crashing the entire C extension (including all_iseqs and exception_message which work fine on those versions). Use have_func in extconf.rb to detect rb_iseq_type at compile time, and wrap the iseq_type function + registration in #ifdef HAVE_RB_ISEQ_TYPE. The Ruby code in code_tracker.rb already handles the missing method via DI.respond_to?(:iseq_type) with a first_lineno fallback. Co-Authored-By: Claude <noreply@anthropic.com>
… require - Add doc comments for rb_iseqw_new and rb_iseqw_to_iseq prototypes in di.c (internal Ruby functions used without documentation) - Add error handling test coverage for backfill_registry: verify logger.debug is called with the error message and telemetry.report is called when DI.current_component is available - Add test coverage for the first_lineno == 0 fallback path when iseq_type is unavailable (Ruby versions without rb_iseq_type) - Add missing require "datadog/di/spec_helper" to iseq_type_spec.rb for consistency with other ext specs - Fix skip message: iseq_type availability depends on rb_iseq_type in the Ruby runtime, not on the DI C extension Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- di.c: Document that rb_iseq_type was added in Ruby 3.1, explain the HAVE_RB_ISEQ_TYPE compile-time guard, and note the fallback path - code_tracker.rb: Replace "first_lineno == 0" YARD doc with full description of both strategies (iseq_type on 3.1+, first_lineno heuristic on older Rubies) and their tradeoffs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The YARD doc claimed the first_lineno == 0 fallback "can match top-level eval iseqs" but this is wrong. InstructionSequence.compile passes first_lineno = 1 (not 0), and require/load passes INT2FIX(0) in Ruby's rb_iseq_new_top/rb_iseq_new_main. Both strategies produce the same result in practice. Verified by reading Ruby 3.0 source (iseq.c lines 813-822): rb_iseq_new_with_opt(ast, name, path, realpath, INT2FIX(0), ...) → ISEQ_TYPE_TOP with first_lineno = 0 And compile path (iseq.c line 1064): rb_iseq_new_with_opt(&ast->body, label, file, realpath, line, ...) → line defaults to INT2FIX(1) for compile/eval Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The raw `bt` ivar on exceptions stores a Thread::Backtrace object, not an Array<String>. Ruby's Exception#backtrace converts it lazily via rb_backtrace_to_str_ary. On newer Ruby versions, `bt` may even be nil with the actual data in `bt_locations` (lazy evaluation). The original implementation returned the raw ivar value, which caused: - Thread::Backtrace returned instead of Array (broke format_backtrace) - nil returned for raised exceptions on newer Ruby (lazy evaluation) Fix by replicating Ruby's conversion logic: 1. If bt is Array (set via set_backtrace), return as-is 2. If bt is Thread::Backtrace, convert via rb_backtrace_to_str_ary 3. If bt is nil, check bt_locations and convert if present rb_backtrace_p and rb_backtrace_to_str_ary are Ruby internal C functions (vm_backtrace.c), not customer code — safe to call from DI instrumentation context. Co-Authored-By: Claude <noreply@anthropic.com>
Style/RedundantBegin: the begin/rescue inside do/end blocks is redundant — the block itself can contain rescue directly. Co-Authored-By: Claude <noreply@anthropic.com>
Add test for the Array code path in the C extension, exercised when Exception#set_backtrace has been called. This covers the RB_TYPE_P(bt, T_ARRAY) early return that wasn't previously tested. Also fix formatting: split keyword args onto separate lines for consistency in probe_notification_builder_spec.rb. Co-Authored-By: Claude <noreply@anthropic.com>
Verify idempotency: calling backfill_registry a second time with the same iseqs doesn't duplicate entries (registry.key? guard). Also verify that a second call with new iseqs adds them without overwriting entries from the first call. Co-Authored-By: Claude <noreply@anthropic.com>
The guard was purely defensive — the C extension is always compiled when DI is active (enforced by environment_supported? in component.rb). The rescue block at the bottom of backfill_registry already catches any exception if file_iseqs fails, making the guard redundant. Co-Authored-By: Claude <noreply@anthropic.com>
The method is called for side effects only. Without the explicit nil, the happy path leaked the synchronize return value and the rescue path leaked the telemetry report return value. Co-Authored-By: Claude <noreply@anthropic.com>
On older Rubies, accessing an uninitialized instance variable via &. produces a warning: "instance variable @current_components not initialized". This triggers loading_spec failures because datadog/di/preload produces unexpected output. The variable is accessed by DI.current_component (called from backfill_registry's error boundary) before any component is added. Initializing to nil at module level suppresses the warning while preserving the existing lazy-init behavior in add_current_component. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RSpec's verify_partial_doubles rejects allow(DI).to receive(:iseq_type) when the method doesn't exist on the module. On Ruby < 3.1, rb_iseq_type is not available so DI.iseq_type is never defined. Fix: conditionally stub iseq_type only when it exists. On older Rubies, let respond_to?(:iseq_type) return false naturally and exercise the first_lineno == 0 fallback path — which is what production does. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pre-loaded test class's iseq can be garbage collected before backfill walks the object space, causing DITargetNotInRegistry. In production, application code is referenced by live constants/methods and survives GC. In the test, the iseq is more ephemeral. Disable GC around activate_tracking! (which calls backfill_registry) to ensure the iseq is still in the object space when all_iseqs runs. Re-enable immediately after. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tions rb_backtrace_p and rb_backtrace_to_str_ary are not exported symbols in Ruby's shared library, causing "undefined symbol: rb_backtrace_p" at runtime on all Ruby versions. Replace with UnboundMethod approach: capture Exception.instance_method(:backtrace) once at init time, then use bind+call to invoke the original C implementation on any exception. This bypasses customer overrides (the UnboundMethod is captured from Exception itself) while using only public Ruby API. Uses bind + call (not bind_call) for Ruby 2.6 compatibility. The UnboundMethod is registered with rb_gc_register_mark_object to prevent GC collection. Co-Authored-By: Claude <noreply@anthropic.com>
rb_backtrace_p and rb_backtrace_to_str_ary are internal Ruby functions
(vm_backtrace.c) that may not be exported as dynamic symbols. The
previous commit declared prototypes manually, which compiled but
failed at runtime with "undefined symbol: rb_backtrace_p" on all
Ruby versions.
Fix: use have_func('rb_backtrace_p') in extconf.rb to detect symbol
availability at compile time. When available, read the bt ivar
directly and convert via rb_backtrace_to_str_ary — no Ruby method
dispatch at all. When unavailable, fall back to calling
Exception#backtrace via an UnboundMethod captured from Exception at
init time, which invokes the original exc_backtrace (error.c)
regardless of subclass overrides.
The bt ivar after raise holds a Thread::Backtrace object. Ruby's
exc_backtrace converts it to Array<String> via rb_backtrace_to_str_ary.
If set via Exception#set_backtrace, bt already holds an Array<String>.
Co-Authored-By: Claude <noreply@anthropic.com>
…ations Remove all C code for exception backtrace (rb_backtrace_p, have_func guard, UnboundMethod fallback in di.c). The conversion functions (rb_backtrace_to_str_ary, rb_backtrace_to_location_ary) are not exported from libruby.so due to missing RUBY_SYMBOL_EXPORT markers in internal/vm.h. Reimplementing via private VM headers is correct but too much work for the gain. Instead, capture Exception.instance_method(:backtrace_locations) as an UnboundMethod at load time. bind(exception).call bypasses subclass overrides — the practical threat model. Does not protect against monkeypatching Exception itself before dd-trace-rb loads. Switch from backtrace (Array<String>) to backtrace_locations (Array<Thread::Backtrace::Location>). DI was regex-parsing the formatted strings back into path/lineno/label — a pointless round-trip. Location objects provide these directly. backtrace_locations available since Ruby 2.6, DI requires 2.6+. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
No wrapper method needed. EXCEPTION_BACKTRACE_LOCATIONS.bind(exc).call is called directly in probe_notification_builder.rb. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_PATTERN Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two root causes: 1. code_tracker_spec.rb: iseq_type was stubbed with and_call_original, but the C function expects a real RubyVM::InstructionSequence, not a test double. Stub returns :top for first_lineno==0, :method otherwise. 2. backfill_integration_spec.rb: The top-level file iseq (first_lineno=0, type=:top) is not referenced by any constant or method after loading. GC could collect it between require_relative (file load time) and the before block's backfill_registry call. Move GC.disable to file level, immediately before require_relative, so the iseq survives until backfill walks the object space. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The ivar is initialized to nil to avoid Ruby 2.6/2.7 warnings. RBS type needs to reflect this. Silence false positive on << after ||= (Steep doesn't track that ||= guarantees non-nil). Co-Authored-By: Claude <noreply@anthropic.com>
Root cause: the top-level (:top) iseq for the test class file has no references after loading completes — only class/method child iseqs survive via BackfillIntegrationTestClass. The previous approach disabled GC at file load time and re-enabled it in the before block after backfill. This protected the first test, but after deactivate_tracking! cleared the registry (the only reference to the iseq), GC could collect it before the next test's backfill_registry walked object space. Fix: capture the top-level iseq in a constant (BACKFILL_TEST_TOP_ISEQ) immediately after loading, before GC can collect it. The constant keeps the iseq alive for the lifetime of the process, so backfill_registry can find it in every test regardless of GC activity. Verified: 0 failures across 8 consecutive full DI suite runs (714 examples each), vs ~20% failure rate before the fix. Co-Authored-By: Claude <noreply@anthropic.com>
Add "Iseq Lifecycle and GC" section to DynamicInstrumentationDevelopment.md covering: iseq types created on file load, which survive GC and why, implications for backfill_registry, and the correct test pattern for keeping top-level iseqs alive across tests. Add cross-reference from code_tracker.rb backfill_registry docstring. Co-Authored-By: Claude <noreply@anthropic.com>
When a file's whole-file (:top) iseq has been garbage collected, per-method iseqs from all_iseqs can still be used to target line probes. This covers 86% of files that were previously untargetable. Changes: - backfill_registry stores per-method iseqs in per_method_registry (grouped by path) instead of discarding them - New iseq_for_line(suffix, line) method tries whole-file iseq first, then searches per-method iseqs for one whose trace_points include the target line - Instrumenter uses iseq_for_line when available, falls back to iseqs_for_path_suffix for compatibility Verified: 37 code_tracker tests pass, lint clean, types clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Loads a test class, GCs the top iseq, then verifies that the backfill finds the surviving method iseq and a line probe can be installed, fired, and captures local variables through it. Precondition checks skip the test if GC didn't collect the top iseq or if the C extension is unavailable. Verified: 3 integration tests pass (install, fire, capture locals). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The throwable now includes a stacktrace array (from the C extension commit). Also update error message assertion for the new raise_if_probe_in_loaded_features format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The raise_if_probe_in_loaded_features now reports whether per-method iseqs exist or not, instead of the generic "not in code tracker registry" message. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Distinguish between "has per-method iseqs but none cover this line" and "has no surviving iseqs at all". Include the target line number in the error. Helps users understand why a line probe failed and whether the file is partially targetable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The method gained line_no and code_tracker parameters in the per-method iseq support commit, but the RBS declaration was not updated. Co-Authored-By: Claude <noreply@anthropic.com>
On Ruby < 3.1, DI.iseq_type is not defined (the C extension gates it with have_func for rb_iseq_type). RSpec's partial double verification rejects stubs for undefined methods, causing 10 failures in the iseq_for_line and per-method iseq tests. Fix: guard iseq_type stubs with `if Datadog::DI.respond_to?(:iseq_type)`, matching the existing pattern in backfill_registry tests (lines 246-255). On Ruby < 3.1, the code under test falls back to first_lineno == 0. Co-Authored-By: Claude <noreply@anthropic.com>
backtrace_locations returns nil when Exception#set_backtrace was called with Array<String> — the VM cannot reconstruct Location objects from formatted strings. This happens in exception wrapping patterns. Add EXCEPTION_BACKTRACE UnboundMethod (same bypass trick) and reinstate the string-parsing format_backtrace_strings as a fallback path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Exception.instance_method(:backtrace).bind(exception).call returns nil when the subclass overrides #backtrace — unlike backtrace_locations, the C implementation of #backtrace does not bypass Ruby-level overrides via UnboundMethod. This is acceptable because EXCEPTION_BACKTRACE is only used as a fallback for the set_backtrace-with-strings case, where no subclass override is involved. Update the test to document the actual behavior and add a comment on the constant explaining the limitation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
MRI's setup_exception (eval.c) calls rb_get_backtrace during raise, which checks rb_method_basic_definition_p for #backtrace overrides. When an override exists, it calls the override, gets a non-nil result, and skips storing the real VM backtrace in @bt and @bt_locations. The C function exc_backtrace then reads @bt (still nil from exc_init) and returns nil. setup_exception does NOT check for #backtrace_locations overrides — only #backtrace. So the EXCEPTION_BACKTRACE_LOCATIONS UnboundMethod works because the real backtrace is always stored in @bt_locations (unless #backtrace is overridden). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…havior Documents why UnboundMethod bypasses backtrace_locations overrides but not backtrace overrides, traced through MRI's setup_exception (eval.c), rb_get_backtrace (error.c), exc_backtrace, and exc_backtrace_locations. Includes code examples for every combination of overrides and set_backtrace, a summary table, and implications for DI's fallback path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- EXCEPTION_BACKTRACE_LOCATIONS: note that overriding #backtrace (not just #backtrace_locations) causes @bt_locations to also be nil - EXCEPTION_BACKTRACE: enumerate all cases — set_backtrace works even with override, only gap is override + raise + no set_backtrace - serialize_throwable: note the unrecoverable case Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
👋 Hey @p-datadog, please fill "Change log entry" section in the pull request description. If changes need to be present in CHANGELOG.md you can state it this way **Change log entry**
Yes. A brief summary to be placed into the CHANGELOG.md(possible answers Yes/Yep/Yeah) Or you can opt out like that **Change log entry**
None.(possible answers No/Nope/None) Visited at: 2026-04-02 23:27:58 UTC |
Typing analysisNote: Ignored files are excluded from the next sections.
|
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 47f8067 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
Remove TEST_OPTIMIZATION_API_KEY secret and set push_to_test_optimization to false while API key usage is transitioning to system-tests. Equivalent to DataDog/dd-trace-java#11043. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bump pinned system-tests ref to 9457e21 (system-tests#6681 — "Skip E2E scenarios and test optim push when DD_API_KEY is missing") and remove the secrets block referencing the deleted DD_API_KEY. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…to base-5540 * origin/di-c-ext-exception-backtrace: (12817 commits) Fix system-tests CI: bump ref to Phase 1 and drop deleted secrets Disable Test Optimization reporting for system-tests Document all backtrace override scenarios in code comments Remove exception backtrace doc (moved to claude-projects) Add doc explaining Exception backtrace internals and UnboundMethod behavior Explain why UnboundMethod doesn't bypass backtrace overrides [APMAPI-1847] Config visibility on all configurations (#5483) Fix EXCEPTION_BACKTRACE test: UnboundMethod does not bypass overrides Fall back to string backtrace when backtrace_locations is nil [🤖] Update System Tests (#5529) Update lib/datadog/core/configuration/config_helper.rb Update lib/datadog/core/environment/identity.rb [🤖] Update System Tests: https://github.com/DataDog/dd-trace-rb/actions/runs/23697412204 Fix Steep: update RBS for format_backtrace and remove BACKTRACE_FRAME_PATTERN Inline exception_backtrace: use constant directly at call site Fix RBS signature: exception_backtrace returns Location not String Replace C exception_backtrace with Ruby UnboundMethod + backtrace_locations Fix undefined symbol: use have_func to gate rb_backtrace_p Fix undefined symbol: use UnboundMethod instead of internal Ruby functions Add set_backtrace test and fix formatting in specs ...
* origin/di-per-method-iseq: (27 commits) Fix iseq_type stub on Ruby < 3.1: guard with respond_to? Fix Steep: update RBS for raise_if_probe_in_loaded_features Improve DITargetNotInRegistry error messages Update remote config test for new error message format Fix throwable integration test to include stacktrace Add integration test for line probe via per-method iseq Support per-method iseqs for line probes on pre-loaded files Document iseq lifecycle and GC interactions for DI backfill Fix backfill integration test: keep top-level iseq alive across tests Fix Steep: allow nil for @current_components Fix StandardRB: add parens to ternary, remove extra blank line Fix backfill_registry test failures Disable GC during backfill integration test to prevent iseq collection Fix backfill_registry tests on Ruby < 3.1 (iseq_type unavailable) Initialize @current_components to suppress Ruby 2.6/2.7 warning Return nil explicitly from backfill_registry Remove respond_to?(:all_iseqs) guard from backfill_registry Add tests for calling backfill_registry twice Fix inaccurate comment: first_lineno == 0 heuristic matches iseq_type Document iseq_type Ruby 3.1 dependency and two-strategy backfill ...
Base Branch
Combines:
Last synced: 2026-04-01