Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Isolate HTTP transports in parallel tests to prevent connection issues #3529

Merged

Conversation

atilsensalduz
Copy link
Contributor

Summary

This PR addresses an issue where parallel tests share the default HTTP transport, leading to intermittent "connection broken" errors due to race conditions in connection reuse.

Root Cause

Tests create their own clients and servers via setup(t), but they unintentionally share Go's default transport when no explicit transport is specified. As a result:

  • server.Close() triggers CloseIdleConnections() on the default transport.
  • This closes all idle connections, affecting other tests running in parallel.
  • If another test is in the middle of using a connection from the shared pool, it encounters a "connection broken" error.

Solution

  • Ensure each test uses an isolated HTTP transport instead of the shared default transport.
  • This prevents unintended connection closures from affecting other tests running concurrently.
  • Improves test stability by eliminating race conditions in connection reuse.

References

atilsensalduz and others added 2 commits March 22, 2025 14:37
- Ensure each test instance uses a dedicated HTTP transport
  instead of sharing the default transport.
- Prevents race conditions caused by CloseIdleConnections()
  closing connections in the shared pool.
- Improves test stability by avoiding intermittent connection broken errors.

References:
- Related discussion: google#3527

Signed-off-by: atilsensalduz <[email protected]>
Copy link

codecov bot commented Mar 22, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.28%. Comparing base (49e644b) to head (5082ea1).
Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3529      +/-   ##
==========================================
- Coverage   91.30%   91.28%   -0.02%     
==========================================
  Files         184      184              
  Lines       16143    16143              
==========================================
- Hits        14739    14736       -3     
- Misses       1230     1231       +1     
- Partials      174      176       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Collaborator

@gmlewis gmlewis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @atilsensalduz!
LGTM.

Awaiting second LGTM+Approval from any other contributor to this repo before merging.

@gmlewis gmlewis added the NeedsReview PR is awaiting a review before merging. label Mar 22, 2025
@Abiji-2020
Copy link
Contributor

@gmlewis can I review it?

@gmlewis
Copy link
Collaborator

gmlewis commented Apr 1, 2025

@gmlewis can I review it?

Absolutely! Thank you, @Abiji-2020! We welcome all contributions.

Copy link
Contributor

@Abiji-2020 Abiji-2020 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Looks Good.
I am just curious that can we reduce the timeout ?

@gmlewis
Copy link
Collaborator

gmlewis commented Apr 2, 2025

✅ Looks Good. I am just curious that can we reduce the timeout ?

Thank you, @Abiji-2020!

The timeout here is sufficiently large such that it should NEVER trigger... and if it ever does, then something is seriously wrong. I think it is good to keep the 30-second timeout so that if it ever does take that long, then it will be pretty obvious and will alert us that it needs investigation as to why. So let's please leave it large like this for now.

@gmlewis gmlewis merged commit 6f329be into google:master Apr 2, 2025
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsReview PR is awaiting a review before merging.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants