Hybrid agent Message Queue Traces and Transactions #1619

lrafeei · 2026-01-03T01:35:53Z

This PR adds support and testing for the following:

Makes use of the sentinel span for trace/span mapping
MessageTraces and MessageTransactions (PRODUCER/CONSUMER kind)

github-actions · 2026-01-03T01:37:21Z

✅MegaLinter analysis: Success

Descriptor	Linter	Files	Fixed	Elapsed time
✅ ACTION	actionlint	7		0.92s
✅ MARKDOWN	markdownlint	7	0	1.39s
✅ PYTHON	ruff	997	26	1.08s
✅ PYTHON	ruff-format	997	31	0.39s
✅ YAML	prettier	15	0	1.61s
✅ YAML	v8r	15		4.81s
✅ YAML	yamllint	15		0.7s

See detailed reports in MegaLinter artifacts

codecov-commenter · 2026-01-03T01:54:50Z

Codecov Report

❌ Patch coverage is 68.29268% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.14%. Comparing base (d71dad1) to head (449a712).
⚠️ Report is 23 commits behind head on develop-hybrid-core-tracing.

Files with missing lines	Patch %	Lines
newrelic/api/opentelemetry.py	68.29%	10 Missing and 3 partials ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           develop-hybrid-core-tracing    #1619      +/-   ##
===============================================================
+ Coverage                        79.93%   80.14%   +0.21%     
===============================================================
  Files                              213      213              
  Lines                            25168    25239      +71     
  Branches                          4001     4023      +22     
===============================================================
+ Hits                             20117    20229     +112     
+ Misses                            3622     3579      -43     
- Partials                          1429     1431       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…queue-traces

hmstepanek

In general implementation looks good. I think the main thing on my mind for this PR is the consumer and producer parts of the agent spec as compared to the tests and some of the comments here-they seem to be a bit at odds with each other but maybe I'm just missing something. I vaguely remember our behavior being different than some of the other agent teams in regards to when we create a trace vs a transaction for message brokers so that might be where my confusion stems from.

hmstepanek · 2026-01-05T17:28:11Z

tests/hybridagent_opentelemetry/test_hybrid_cross_agent.py

-    with BackgroundTask(application, name="Foo"):
-        with tracer.start_as_current_span(name="Bar", kind=otel_api_trace.SpanKind.INTERNAL):
-            try:
+    with pytest.raises(Exception):


Suggested change

with pytest.raises(Exception):

with pytest.raises(Exception, match=r"Test exception message"):

hmstepanek · 2026-01-05T17:34:09Z

tests/hybridagent_fastapi/test_otel_application.py

-        scoped_metrics=[(f"Function/{transaction_name}", 1)],
-        rollup_metrics=_test_application_rollup_metrics + [(f"Function/{transaction_name}", 1)],
+        scoped_metrics=[(f"Function/{transaction_name} http send", 2)],
+        rollup_metrics=_test_application_rollup_metrics + [(f"Function/{transaction_name} http send", 2)],


Fixup for megalinter:

Suggested change

rollup_metrics=_test_application_rollup_metrics + [(f"Function/{transaction_name} http send", 2)],

rollup_metrics=[(f"Function/{transaction_name} http send", 2), *_test_application_rollup_metrics],

newrelic/api/opentelemetry.py

hmstepanek · 2026-01-05T17:57:16Z

newrelic/api/opentelemetry.py

+            # While this OTel span exists it will not be explicitly
+            # translate to a NR trace.  This may occur during the
+            # creation of a Transaction, which will create the root
+            # span.  This may also occur during special cases, such


Question about this-the spec says the following:

Creating a span with a span kind of SpanKind.CONSUMER will start a OtherTransaction/* transaction
and create a corresponding segment.

If the CONSUMER span occurs within an already existing NR transaction, it will be included in
the transaction trace.

Wouldn't the above mean that a nr trace should be created on multiple calls to the consumer?

Yep! For the rest of the message queues that is what we do. For Kafka, we ended up treating the consumer as a transaction and the producer as a trace. But yes, I have another PR for the rest of the queues that tweaks this logic.

hmstepanek · 2026-01-05T18:13:50Z

newrelic/api/opentelemetry.py

-        if not nr_trace or (nr_trace and getattr(nr_trace, "end_time", None)):
+        # Check to see if New Relic trace ever existed
+        # or, if it does, that trace has already ended
+        if getattr(self.nr_trace, "end_time", None):


This isn't quite the same behavior as the previous. In the previous if the self.nr_trace was None it would also return. Do we want to match the previous behavior still?

Suggested change

if getattr(self.nr_trace, "end_time", None):

if not self.nr_trace or getattr(self.nr_trace, "end_time", None):

Oof, not sure how that piece of the logic got removed--that should have stayed in there...

hmstepanek · 2026-01-05T18:20:54Z

newrelic/api/opentelemetry.py

+        # Message specific attributes
+        if self.attributes.get("messaging.system"):
+            destination_name = self.attributes.get("messaging.destination")
+            self.nr_transaction.destination_name = destination_name


Suggested change

self.nr_transaction.destination_name = destination_name

Oh, I remember why I stored it as a separate variable instead of directly putting it in as a transaction attribute: I also use that name in the Kafka Nodes metric a few lines below.

Oh--oops missed that. Ok fine as is then!

newrelic/api/opentelemetry.py

hmstepanek · 2026-01-05T18:42:36Z

newrelic/api/opentelemetry.py

+        # NOTE: Sentinel, MessageTrace, DatastoreTrace, and
+        # ExternalTrace types do not have a name attribute.
        self._name = name
        if hasattr(self, "nr_trace") and hasattr(self.nr_trace, "name"):


It looks like we are missing coverage for this function.

hmstepanek · 2026-01-05T18:43:29Z

newrelic/api/opentelemetry.py

-        if not nr_trace or (nr_trace and getattr(nr_trace, "end_time", None)):
+        # Check to see if New Relic trace ever existed
+        # or, if it does, that trace has already ended
+        if getattr(self.nr_trace, "end_time", None):


It looks like we are missing coverage in the tests for this case.

hmstepanek · 2026-01-05T18:52:35Z

tests/hybridagent_kafkapython/test_producer.py

+            send_producer_message()
+
+    test()
+


Wondering if we should have a test here to validate that a transaction and trace are not created for a producer in the case that the span exists outside a transaction?

Creating a span with a span kind of SpanKind.PRODUCER will not start a NR transaction.

If the PRODUCER span occurs within an already existing NR transaction, a segment will
be created and it will be included in the transaction trace.

That's fair. I'll put in a test for the first scenario and the second scenario I will use an existing consumer test to prove that this is the case

Co-authored-by: Hannah Stepanek <[email protected]>

lrafeei added 2 commits January 2, 2026 17:28

Change span logic to use sentinels

0210eaf

Kafka-python support (PRODUCER/CONSUMER)

d3a682e

lrafeei changed the title ~~Hybrid agent message queue traces~~ Hybrid agent Message Queue Traces and Transactions Jan 3, 2026

mergify bot added the tests-failing Tests failing in CI. label Jan 3, 2026

hmstepanek marked this pull request as ready for review January 5, 2026 17:25

hmstepanek requested a review from a team as a code owner January 5, 2026 17:25

Merge branch 'develop-hybrid-core-tracing' into hybrid-agent-message-…

0dacf19

…queue-traces

hmstepanek requested changes Jan 5, 2026

View reviewed changes

lrafeei and others added 2 commits January 5, 2026 12:00

Apply suggestions from code review

449a712

Co-authored-by: Hannah Stepanek <[email protected]>

Reviewer suggestions

9d67cc8

lrafeei requested a review from hmstepanek January 5, 2026 21:02

Use json.loads instead of eval

c30a0e4

lrafeei force-pushed the hybrid-agent-message-queue-traces branch from 207bf9c to c30a0e4 Compare January 5, 2026 21:21

[MegaLinter] Apply linters fixes

14271d6

hmstepanek approved these changes Jan 6, 2026

View reviewed changes

hmstepanek merged commit b3dc905 into develop-hybrid-core-tracing Jan 6, 2026
6 checks passed

hmstepanek deleted the hybrid-agent-message-queue-traces branch January 6, 2026 18:10

mergify bot removed the tests-failing Tests failing in CI. label Jan 6, 2026

	with pytest.raises(Exception):
	with pytest.raises(Exception, match=r"Test exception message"):

	rollup_metrics=_test_application_rollup_metrics + [(f"Function/{transaction_name} http send", 2)],
	rollup_metrics=[(f"Function/{transaction_name} http send", 2), *_test_application_rollup_metrics],

	if getattr(self.nr_trace, "end_time", None):
	if not self.nr_trace or getattr(self.nr_trace, "end_time", None):

Hybrid agent Message Queue Traces and Transactions #1619

Hybrid agent Message Queue Traces and Transactions #1619

Uh oh!

Conversation

lrafeei commented Jan 3, 2026

Uh oh!

github-actions bot commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅MegaLinter analysis: Success

Uh oh!

codecov-commenter commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hmstepanek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Jan 3, 2026 •

edited

Loading

codecov-commenter commented Jan 3, 2026 •

edited

Loading