Support different http backends?

[peterHoburg]

Over the last 2 years a lot of people have been flagging issues with HTTPx when it comes to scaling concurrent calls. Specifically around pooling, async, and parallel requests.

https://github.com/encode/httpx/issues/3215

There have been multiple attempts to patch it, including a fix that was merged then reverted by the maintainer. This reddit thread talks about some issues. https://www.reddit.com/r/Python/comments/1qdel5w/comment/nzp86y6/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

My main concern with HTTPx is that the maintainer does not seem concerned about performance in large(ish) production workloads, and performance work is being neglected.

All that being said, I mostly use HTTPx, but have also used aiohttp, and requests.
Another interesting (new) contender is https://github.com/MarkusSintonen/pyreqwest . Python wrapper on the very good rust reqwest lib.

I am not pushing to get off HTTPx, or stay on it, just start a discussion for 2.0 (this type of change would need a major bump IMO)

[phalt]

I have considered a way to swap backends. What I would do is build the abstraction and have it support httpx, then let others contribute other backend adaptions. The surface area would be tiny.

So yes I agree. Let's include this with the 2.0 rollout.

[phalt]

FWIW I use httpx at work, with massive traffic, on one of the biggest Django sites in the world. I've not observed performance issues. But that is another division.

I agree that offering the option of a swappable network backend is worth doing.

[phalt]

I have implemented basic support here: #175

It needs a thorough review. One thing I'd really like for someone to do is pull down the branch and test with a second backend. aiohttp is quite different in their api, so something quite different as opposed to something very httpx-like is preferable.

[peterHoburg]

I left 2 comments on the PR, but you had already merged it :( Sorry it took me a while to get around to looking at it.

[phalt]

Shipped in 1.7.0