[Bug]: Cannot get past 50 RPS

[vutrung96]

What happened?

I have OpenAI tier 5 usage, which should give me 30,000 RPM = 500 RPS with "gpt-4o-mini". However I struggle get past 50 RPS.

The minimal replication:

from litellm import acompletion

tasks = [acompletion(
    model="gpt-4o-mini",
    messages=[
      {"role": "system", "content": "You're an agent who answers yes or no"},
      {"role": "user", "content": "Is the sky blue?"},
    ],
) for i in range(2000)]

I only get 50 items/second as opposed to ~500 items/second when sending raw HTTP requests.

Relevant log output

 16%|█████████████████████▌                                                                                                                 | 320/2000 [00:09<00:40, 41.49it/s]

Twitter / LinkedIn details

No response

[ishaan-jaff]

hi @vutrung96 looking into this, how do you get the % complete log output ?

[vutrung96]

Hi @ishaan-jaff I was just using tqdm

[CharlieJCJ]

Hi @ishaan-jaff , any updates on this, also facing this issue!

[ishaan-jaff]

hi @vutrung96 @CharlieJCJ do you see the issue on litellm.router too ? https://docs.litellm.ai/docs/routing

It would help me if you could test with litellm router too

[RyanMarten]

Hi @ishaan-jaff
We tracked down the root cause of the issue.

Litellm uses the official OpenAI python client

client: Optional[Union[OpenAI, AsyncOpenAI]] = None,

The official OpenAI client has performance issues with high numbers of concurrent requests due to issues in httpx

• httpx client has very poor performance for concurrent requests compared to aiohttp openai/openai-python#1596

The issues in httpx are due to a number of factors related to anyio vs asyncio

• Improve async performance. encode/httpx#3215

Which are addressed in the open PRs below

• Use asyncio synchronization instead of anyio encode/httpcore#922
• Connection pool optimization: move socket polling from expiry checks to connection usage encode/httpcore#928
• Connection pool optimization: reduce connection maintenance loops complexity encode/httpcore#929
• Add asyncio backend encode/httpcore#930
• Use AnyIO with fast_acquire synchronization  encode/httpcore#953

We saw this when implementing litellm as the backend for our synthetic data engine

• Add LiteLLM+instructor (for structured output) backend for curator bespokelabsai/curator#141

When using our own openai client (with aiohttp instead of httpx) we saturate the highest rate limits (30,000 requests per minute on gpt-4o-mini tier 5). When using litellm, the performance issues cap us well under the highest rate limit (200 queries per second - 12,000 requests per minute).

[ishaan-jaff]

@RyanMarten you are right ! just ran a load test to confirm. The right is with aiohttp it's 10x more RPS

[ishaan-jaff]

@RyanMarten started work on this

#7514

• added a new custom_llm_provider=aiohttp_openai that uses aiohttp for calling logic

@RyanMarten, @vutrung96 and @CharlieJCJ can y'all help us test this change as we start rolling it out ?

As of now we just added support for non-streaming. I can let you know once streaming support is added too

[RyanMarten]

@ishaan-jaff Thanks for creating a PR for this! We can certainly help test the change 😄. I'll run a benchmarking test with model=aiohttp_openai/gpt-4o-mini

Our use-case on non-streaming so that shouldn't be a problem.

[RyanMarten]

Here is our benchmarking using the curator request processor and viewer (with different backends). I see that this was released in https://github.com/BerriAI/litellm/releases/tag/v1.56.8. I upgraded my litellm version to the latest poetry add litellm@latest which is 1.57.0.

from bespokelabs.curator import LLM
from datasets import Dataset

dataset = Dataset.from_dict({"prompt": ["write me a poem"] * 100_000})

(1) our own aiohttp backend

llm = LLM(
    prompt_func=lambda row: row["prompt"],
    model_name="gpt-4o-mini",
    backend="openai",
)
dataset = llm(dataset)

(2) default litellm backend

llm = LLM(
    prompt_func=lambda row: row["prompt"],
    model_name="gpt-4o-mini",
    backend="litellm",
)
dataset = llm(dataset)

(3) litellm backend with aiohttp_openai

llm = LLM(
    prompt_func=lambda row: row["prompt"],
    model_name="aiohttp_openai/gpt-4o-mini",
    backend="litellm",
)
dataset = llm(dataset)

For some reason I'm not seeing an improvement in performance