ref(otel): Add `ClientOptions.Instrumenter` #679

costela · 2023-07-28T22:46:29Z

This small PR is a first attempt to fix the perceived issue in #678.

It extends sentry.ClientOptions with the Instrumenter parameter, analogous to other language SDKs. This can be optionally set to "otel" or "sentry" (the default).

This new option is then respected by middlewares to optionally skip creating a sentry transaction, allowing the consumer to only use "pure" otel for traces.

Considered alternatives:

Another option would be making the transaction created by the otel span processor visible to the middlewares. However, since the span processor cannot easily modify the request context, it becomes necessary to add additional helpers (e.g. a new "adapter" middleware) with the single task of injecting the otel-created transaction into the request context.

Fixes: #678 #788

codecov · 2023-07-31T13:11:23Z

Codecov Report

Attention: Patch coverage is 97.43590% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 81.20%. Comparing base (255c172) to head (8249502).

Files	Patch %	Lines
otel/span_processor.go	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #679      +/-   ##
==========================================
+ Coverage   80.89%   81.20%   +0.30%     
==========================================
  Files          48       48              
  Lines        4717     4730      +13     
==========================================
+ Hits         3816     3841      +25     
+ Misses        763      754       -9     
+ Partials      138      135       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

costela · 2023-07-31T13:27:28Z

The windows test failure seems unrelated? Maybe flaky?

As for the coverage decrease, I could add some tests for this case if the general approach seems reasonable.

cleptric · 2023-07-31T13:31:09Z

Yep, Windows tests are sadly a bit flaky 🙁
More tests are always welcome. I didn't look into how other SDKs handle this yet, did you maybe check https://github.com/getsentry/sentry-javascript already?

costela · 2023-08-16T08:42:20Z

Sorry for the delay; vacation got in the way 😉

My javascript is pretty rusty (no 🦀 pun intended), but AFAICT the js integration works slightly differently. It always uses an existing span context (similarly to the proposed change in this PR), but it also has an extra instrumenter option, unlike the go integration (and this PR, for that matter).

The extra option seems a bit error-prone, compared to the sample-func approach in this PR: the code has to remember to check against known values, instead of explicitly passing exactly what we want it to do. But that's just my subjective (and superficial!) take on it.

costela · 2023-08-23T08:22:35Z

I've reworked the PR with a slightly different solution. Now instead of trying to use a sampler, we address the issue at the middleware level.

The problem - as I curently understand it - is related to middleware call-ordering: we want otel to be called first, to respect potential sampling decisions from a remote caller. That sampling decision should be passed to the sentry world, which was already done in the otel span processor, but was not working reliably (fixed in this PR).

After the otel middleware, we still want to use the sentry middleware (at least for event tracking and recovery), but at that point, the existing sentry span created in the span processor above is not in the context, so the sentry middleware happily creates a new one (and without a parent).
To avoid this, we need to tell the sentry middleware to also look for an existing otel span and get the sentry span from that.

Does that make sense?

costela · 2023-08-24T11:41:15Z

Ok, third time is a charm!

The sentry.SpanOption doesn't quite work, because it can't modify it's received context. So we have to go with yet another middleware, which sits as an adapter between otel and sentryhttp.Handler.
It sets the transaction created via the span processor in the context passed to sentryhttp.Handler, so that it detects its existence and can react accordingly.

PTAL

costela · 2023-08-28T10:26:28Z

@cleptric can you please allow the tests to run? 🙏

costela · 2023-09-19T09:39:14Z

FYI: 🔁 rebased on master, please unlock tests 🙏

⚠️ this is only to allow using the forked version while getsentry#679 is open

cleptric · 2023-10-04T09:38:35Z

Sorry that it took us a while to get back to you.

When using Otel, you shouldn't use any Sentry performance-related features. This is why we added the instrumenter option on other SDKs, which would disable all Sentry instrumentation. In Go, we opted not to add this option, given the manual nature of performance tracing.

So instead of using Sentry's net/http instrumentation, you would use https://github.com/open-telemetry/opentelemetry-go-contrib.

Our propagator considers incoming trace headers, so distributed tracing also works.

costela · 2023-10-04T12:08:15Z

So instead of using Sentry's net/http instrumentation, you would use https://github.com/open-telemetry/opentelemetry-go-contrib.

So you mean skipping the http middleware? But then we'd also throw out error/panic tracking?

If that's the intended way to work with OTEL, then maybe I should change this PR to instead add an option to sentryhttp.Options to skip traces?

cleptric · 2023-10-04T12:19:29Z

This is a good point about panic tracking I haven't thought about.

IMO, we could wrap this code https://github.com/getsentry/sentry-go/blob/master/http/sentryhttp.go#L97-L112 into an if condition that checks for ClientOptions.EnableTracing, so no transactions are created.

This might make it a bit easier than adding a new integration option.

costela · 2023-10-04T12:39:00Z

@cleptric you mean disabling ClientOptions.EnableTracing? I think that alone won't quite work either: then all transactions get dropped (even if the OTEL propagator decides to sample them).

We could maybe refactor the sampling logic to respect sample decisions even if EnableTracing is false, but I fear that might be too surprising to users.

In the end we want sentryhttp.Handler to only start a transaction if none has been started already (e.g. by the otel propagator), but unfortunately I haven't found a nice way to pass that informatino along. The propagator can't modify the request context and they are otherwise unaware of each other. That's why I ended up with the current suggestion in this PR. 😕

It seems to me we have two options:

we either need to glue the sentry OTEL propagator to the sentryhttp middleware (e.g. via the proposed middleware in this PR)
or we add an explicit option to the sentryhttp middleware to not start transactions

cleptric · 2023-10-04T13:02:48Z

Ok, we might need to add the instrumenter option to control the behaviour. If Otel is active, no transactions/spans should be created by the Sentry SDK.

greywolve · 2023-10-04T13:04:54Z

I think adding the instrumenter option also seems to be consistent with the other sdks.

costela · 2023-10-04T15:55:12Z

@cleptric @greywolve now this PR has an attempted implementation of the new TracesInstrumenter option. PTAL 🙏

cleptric

Added some comments. This also needs tests.

cleptric · 2023-10-04T19:05:20Z

client.go

@@ -133,6 +133,9 @@ type ClientOptions struct {
 	TracesSampleRate float64
 	// Used to customize the sampling of traces, overrides TracesSampleRate.
 	TracesSampler TracesSampler
+	// Which instrumentation to use for tracing. Either "sentry" or "otel" are supported.
+	// Setting this to "otel" will ignore TracesSampleRate and TracesSampler and assume sampling is performed by otel.
+	TracesInstrumenter string


We should call it Instrumenter as in other SDKs.

Sure thing! Done! PTAL

cleptric · 2023-10-04T19:16:32Z

client.go

+	switch options.TracesInstrumenter {
+	case "":
+		options.TracesInstrumenter = "sentry"
+	case "sentry":
+		// noop
+	case "otel":
+		// sampling is performed by the OpenTelemetry SDK
+		options.TracesSampleRate = 1.0
+		options.TracesSampler = nil
+	default:
+		return nil, fmt.Errorf("invalid value for TracesInstrumenter (supported are 'sentry' and 'otel'): %q", options.TracesInstrumenter)
+	}
+


I'm not sure if we should do this. I would rather have people define this explicitly in their options.

you mean the TracesSampleRate and TracesSampler? Should we then error if the values are set to anything else? Doing neither seems a bit error-prone: Instrumenter: "otel" + TracesSampleRate: 0 (default) would ignore otel sampling decisions and never send traces.

Are you sure it does this? The span processor actually takes the Otel sampling decision into account, taking priority over the traces sample rate and traces sampler.

I believe so. That might be a slightly different issue, though. One I had addressed in a previous commit.
The current span processor considers only the sampled decision of the parent, so it only actually respects it if the span is part of an incoming trace. If it's a new trace, without a parent, the otel span might be sampled, but the sentry span remains at SampleUndefined, which then means we fall back to the client-wide sample rate.

The current test doesn't catch it because both the sentry client and the otel stack are initialized with "always trace", so they both arrive at the same decision, even though the processor isn't doing anything.

I just added a commit with the fixed test and the fixed code. PTAL.

friendly ping 😇

I'll take a look later this week.

We made similar changes for node.js in getsentry/sentry-javascript#9203, but I need to dive into this whole topic a bit deeper to give you actionable feedback.

friendly ping 😇 (I know these things get lower prio; just wanna avoid it getting forgotten)

Yeah, sorry for that! I had to get some stuff shipped over on the PHP side of things, but the next few weeks will be again Go focus time for me.

costela · 2023-10-12T17:39:44Z

I want to add another perspective to this MR, that may mean the ClientOptions.Instrumenter is not enough: some client SDKs (e.g. sentry-cocoa) do not seem to support opentelemetry at all. These transactions would be lost when speaking to a sentry+otel service. The service only reads otel headers, the client only sends sentry headers.

The suggestion with the additional middleware would not have this problem: incoming requests could use either sentry or otel propagation, the service code can leverage existing otel integrations (e.g. for pgx; shameless plug 🙈)

~~Unless sentry plans to support otel propagation on client sdks, it may be necessary to make the service side accept both. Or am I overlooking something?~~

(nevermind, missed the propagator code, sorry for the noise 🙇)

cleptric · 2023-10-13T02:54:22Z

We actually extract the Sentry trace header, see

https://github.com/getsentry/sentry-go/blob/master/otel/propagator.go#L77-L95

costela · 2023-10-13T07:36:16Z

@cleptric ah, you're right of course. Appologies for the noise; disregard the last comment.

client.go

greywolve · 2023-11-09T02:56:09Z

LGTM, but I'll let @cleptric give the final thumbs up :)

costela · 2023-11-21T13:44:14Z

hi!
just a little bi-monthly rebase + reminder about this PR 😇

costela · 2023-12-18T08:47:22Z

maybe I can get this PR as a christmas present? 😇 🎅 🎄

(just rebased 🔁)

costela · 2024-02-23T08:34:37Z

Rebased again.

Any chance a paying customer can get some feedback? Even if this isn't the right solution?

cleptric · 2024-02-26T18:03:05Z

It's still on our radar!

rodolfoBee · 2024-06-05T05:39:00Z

@cleptric customer has requested an update on this PR. Is there any?

This is a fork based on this branch: getsentry#679

This enables instrumentation code (sentryhttp, sentrygin, etc) to skip trace instrumentation while still performing panic recovery.

Co-authored-by: Oliver Powell <[email protected]>

aldy505 · 2024-08-20T00:58:46Z

@cleptric any updates for this?

cleptric added the Type: Bug label Jul 31, 2023

tonyo added the Status: In Progress label Aug 16, 2023

costela force-pushed the respect-otel-sampling branch 2 times, most recently from db4a1ea to 8e878cd Compare August 23, 2023 08:13

costela mentioned this pull request Aug 23, 2023

[otel] integration should sync sampling decision to/from otel in the "parentless" case #678

Closed

costela force-pushed the respect-otel-sampling branch 2 times, most recently from 1736444 to 9b18d57 Compare August 24, 2023 11:34

costela force-pushed the respect-otel-sampling branch from 9b18d57 to af3cd83 Compare September 19, 2023 09:38

costela added a commit to exaring/sentry-go that referenced this pull request Sep 25, 2023

chore: rename module to fork

e112715

⚠️ this is only to allow using the forked version while getsentry#679 is open

costela force-pushed the respect-otel-sampling branch from af3cd83 to 98709e9 Compare October 4, 2023 15:53

cleptric reviewed Oct 4, 2023

View reviewed changes

cleptric changed the title ~~feat: respect otel sampling~~ ref(otel): Add ClientOptions.Instrumenter Oct 4, 2023

costela force-pushed the respect-otel-sampling branch 2 times, most recently from a775255 to 0c4ac8e Compare October 5, 2023 13:14

costela requested a review from cleptric October 5, 2023 15:56

costela force-pushed the respect-otel-sampling branch from 0c4ac8e to b48adb9 Compare October 9, 2023 12:58

costela force-pushed the respect-otel-sampling branch 2 times, most recently from b9e108b to d8a5714 Compare October 23, 2023 15:30

greywolve reviewed Nov 8, 2023

View reviewed changes

client.go Outdated Show resolved Hide resolved

costela requested a review from greywolve November 8, 2023 16:51

costela force-pushed the respect-otel-sampling branch from 27db82c to d389e8a Compare November 21, 2023 13:42

costela force-pushed the respect-otel-sampling branch from d389e8a to befd5e5 Compare December 18, 2023 08:46

costela force-pushed the respect-otel-sampling branch from befd5e5 to 8249502 Compare February 22, 2024 08:47

costela mentioned this pull request Feb 23, 2024

[otel] breaks sampling when used together with sentry middleware #788

Open

costela force-pushed the respect-otel-sampling branch 2 times, most recently from 2e505ec to 49e1741 Compare July 10, 2024 12:54

costela added a commit to exaring/sentry-go that referenced this pull request Jul 10, 2024

chore: fork

eecc552

This is a fork based on this branch: getsentry#679

costela force-pushed the respect-otel-sampling branch from 49e1741 to 92bb309 Compare August 13, 2024 09:12

costela added a commit to exaring/sentry-go that referenced this pull request Aug 13, 2024

chore: fork

2a50e53

This is a fork based on this branch: getsentry#679

costela and others added 4 commits August 19, 2024 22:26

feat: add TracesInstrumenter option

0a318d5

This enables instrumentation code (sentryhttp, sentrygin, etc) to skip trace instrumentation while still performing panic recovery.

test: cover new instrumenter client option

b77e457

fix(otel): also respect sampling on parentless spans

863d340

fix: report correct option in error

394d48e

Co-authored-by: Oliver Powell <[email protected]>

costela force-pushed the respect-otel-sampling branch from 92bb309 to 394d48e Compare August 19, 2024 20:26

ref(otel): Add ClientOptions.Instrumenter #679

Are you sure you want to change the base?

ref(otel): Add ClientOptions.Instrumenter #679

Conversation

costela commented Jul 28, 2023 • edited Loading

Considered alternatives:

codecov bot commented Jul 31, 2023 • edited Loading

Codecov Report

costela commented Jul 31, 2023 • edited Loading

cleptric commented Jul 31, 2023

costela commented Aug 16, 2023

costela commented Aug 23, 2023

costela commented Aug 24, 2023

costela commented Aug 28, 2023

costela commented Sep 19, 2023

cleptric commented Oct 4, 2023

costela commented Oct 4, 2023

cleptric commented Oct 4, 2023

costela commented Oct 4, 2023 • edited Loading

cleptric commented Oct 4, 2023 • edited Loading

greywolve commented Oct 4, 2023

costela commented Oct 4, 2023

cleptric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costela Oct 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costela commented Oct 12, 2023 • edited Loading

cleptric commented Oct 13, 2023

costela commented Oct 13, 2023

greywolve commented Nov 9, 2023

costela commented Nov 21, 2023

costela commented Dec 18, 2023

costela commented Feb 23, 2024

cleptric commented Feb 26, 2024

rodolfoBee commented Jun 5, 2024

aldy505 commented Aug 20, 2024

ref(otel): Add `ClientOptions.Instrumenter` #679

ref(otel): Add `ClientOptions.Instrumenter` #679

costela commented Jul 28, 2023 •

edited

Loading

codecov bot commented Jul 31, 2023 •

edited

Loading

costela commented Jul 31, 2023 •

edited

Loading

costela commented Oct 4, 2023 •

edited

Loading

cleptric commented Oct 4, 2023 •

edited

Loading

costela Oct 5, 2023 •

edited

Loading

costela commented Oct 12, 2023 •

edited

Loading