Validate fix for flaky: shutdown 'executes only once'#5586
Draft
Validate fix for flaky: shutdown 'executes only once'#5586
Conversation
Hypothesis: the test fails when the HTTP round-trip to the agent takes longer than DEFAULT_SHUTDOWN_TIMEOUT (1 second). The worker thread is mid-HTTP-call when shutdown! calls join(1), the join times out, and traces_flushed is still 0 when the assertion runs. This commit forces the race condition by stubbing the transport's send_traces to add a 2-second sleep, exceeding the 1-second shutdown timeout. The test should fail deterministically in CI. Co-Authored-By: Claude <noreply@anthropic.com>
Root cause: the test asserts traces_flushed: 1 immediately after concurrent shutdown! calls join. When the HTTP round-trip to the agent takes longer than DEFAULT_SHUTDOWN_TIMEOUT (1s), the worker thread's join times out and the flush is still in-progress. The assertion sees traces_flushed: 0 and fails. Wait for traces_flushed >= 1 AFTER the shutdown threads join, not before. This preserves the test's coverage: the concurrent shutdowns still race with an in-flight flush (the buffer has data when shutdowns begin), so the shutdown guard and worker stop logic are exercised under contention. The wait just gives the orphaned worker thread time to complete its HTTP call before the assertion checks final stats. Co-Authored-By: Claude <noreply@anthropic.com>
* reproduce-flaky-shutdown-executes-only-once: Add reproducer for flaky shutdown 'executes only once' test
* fix-flaky-shutdown-executes-only-once: Fix flaky shutdown 'executes only once' test
p-datadog
added
the
AI Generated
Contributor
|
✅ Tests 🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 8434186 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Validates that the fix neutralizes the forced failure. This branch merges:
If CI passes, the fix addresses the exact failure mode the reproducer forces.
If CI fails, the fix does not solve the problem.
Do not merge — this PR exists for validation only.
Change log entry
None.
How to test the change?
CI passes = fix works against the forced failure. CI fails = fix is insufficient.