Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClientRequest's constructor makes a blocking IO call #10435

Open
1 task done
cbornet opened this issue Feb 8, 2025 · 4 comments
Open
1 task done

ClientRequest's constructor makes a blocking IO call #10435

cbornet opened this issue Feb 8, 2025 · 4 comments
Labels
Milestone

Comments

@cbornet
Copy link
Contributor

cbornet commented Feb 8, 2025

Describe the bug

In ClientRequest's constructor there is a call to update_auth which in turn calls netrc_from_env.
netrc_from_env is blocking as it reads the .netrc file from the FS.
So instanciating ClientRequest in the event loop is blocking the loop.

This was detected by BlockBuster in #10433

To Reproduce

N/A

Expected behavior

I would expect ClientRequest to not block the asyncio loop.

Logs/tracebacks

N/A

Python Version

3.11

aiohttp Version

HEAD

multidict Version

6.1.0

propcache Version

0.2.1

yarl Version

1.18.3

OS

macOS

Related component

Client

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct
@cbornet cbornet added the bug label Feb 8, 2025
@cbornet
Copy link
Contributor Author

cbornet commented Feb 8, 2025

Caching the result of netrc_from_env (eg with @functools.cache) could be a first step to minimize the impact.
But in that case if the file is modified, it won't be reloaded (unless there is a blocking check of the modification time, which is still not as bad as reading the file).

@Dreamsorcerer Dreamsorcerer added this to the 4.0 milestone Feb 8, 2025
@Dreamsorcerer
Copy link
Member

We can potentially put a proper fix into v4 and mitigations into v3, if necessary.

@asvetlov
Copy link
Member

asvetlov commented Feb 8, 2025

How long is the call blocks?
Do you have numbers?
I expect that the filesystem cache works pretty well in this case.

@cbornet
Copy link
Contributor Author

cbornet commented Feb 10, 2025

I don't have numbers for how long it blocks. This file is usually small, so it will probably not block for long and page cache should indeed work pretty well.
I understand the tradeoff between making a breaking change and doing a small optimization.
Maybe the best is to do nothing or just add some runtime cache with an os.stat check (a bit like what is done for linecache).
LMK what you decide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants