Draft
Conversation
Root cause: the test asserts traces_flushed: 1 immediately after concurrent shutdown! calls join. When the HTTP round-trip to the agent takes longer than DEFAULT_SHUTDOWN_TIMEOUT (1s), the worker thread's join times out and the flush is still in-progress. The assertion sees traces_flushed: 0 and fails. Wait for traces_flushed >= 1 AFTER the shutdown threads join, not before. This preserves the test's coverage: the concurrent shutdowns still race with an in-flight flush (the buffer has data when shutdowns begin), so the shutdown guard and worker stop logic are exercised under contention. The wait just gives the orphaned worker thread time to complete its HTTP call before the assertion checks final stats. Co-Authored-By: Claude <noreply@anthropic.com>
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 0e5fc33 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes flaky test
spec/datadog/tracing/integration_spec.rb:592— the shutdown "executes only once" integration test.Motivation:
Flaky test reported in PR #5581. The test intermittently fails with
traces_flushed: 0under CI load.Failure: The assertion
expect(stats).to include(traces_flushed: 1)fails withtraces_flushed: 0.Root cause: The test asserts
traces_flushed: 1immediately after concurrentshutdown!calls join.shutdown!callsWriter#stop, which callsAsyncTransport#stop, which joins the worker thread with a 1-second timeout (DEFAULT_SHUTDOWN_TIMEOUT). When the HTTP round-trip to the agent takes longer than 1 second under CI load, the join times out. The worker thread is still alive completing the HTTP call, but@traces_flushedhasn't been incremented yet. The assertion fires and sees 0.Fix: Add
try_wait_until { tracer.writer.stats[:traces_flushed] >= 1 }AFTER the shutdown threads join, not before them. This preserves the test's coverage: the concurrent shutdowns still race with an in-flight flush (the buffer has data when shutdowns begin), so the shutdown guard and worker stop logic are exercised under real contention. The wait just gives the orphaned worker thread time to complete its HTTP call before the assertion checks final stats.This is deliberately different from PR #5581's approach (which waits BEFORE shutdowns). Waiting before drains the buffer, so all shutdown flushes are no-ops — making
traces_flushed: 1tautological regardless of whether the shutdown guard works. Waiting after preserves the race between shutdowns and the flush.Change log entry
None.
How to test the change?
Integration test validated via the three-PR reproducer/fix/validation pattern: