litellm.md

Using any model via LiteLLM

!!! note

The LiteLLM integration is in beta. You may run into issues with some model providers, especially smaller ones. Please report any issues via [Github issues](https://github.com/openai/openai-agents-python/issues) and we'll fix quickly.

LiteLLM is a library that allows you to use 100+ models via a single interface. We've added a LiteLLM integration to allow you to use any AI model in the Agents SDK.

Setup

You'll need to ensure litellm is available. You can do this by installing the optional litellm dependency group:

pip install "openai-agents[litellm]"

Once done, you can use [LitellmModel][agents.extensions.models.litellm_model.LitellmModel] in any agent.

Example

This is a fully working example. When you run it, you'll be prompted for a model name and API key. For example, you could enter:

• openai/gpt-4.1 for the model, and your OpenAI API key
• anthropic/claude-3-5-sonnet-20240620 for the model, and your Anthropic API key
• etc

For a full list of models supported in LiteLLM, see the litellm providers docs.

from __future__ import annotations

import asyncio

from agents import Agent, Runner, function_tool, set_tracing_disabled
from agents.extensions.models.litellm_model import LitellmModel

@function_tool
def get_weather(city: str):
    print(f"[debug] getting weather for {city}")
    return f"The weather in {city} is sunny."


async def main(model: str, api_key: str):
    agent = Agent(
        name="Assistant",
        instructions="You only respond in haikus.",
        model=LitellmModel(model=model, api_key=api_key),
        tools=[get_weather],
    )

    result = await Runner.run(agent, "What's the weather in Tokyo?")
    print(result.final_output)


if __name__ == "__main__":
    # First try to get model/api key from args
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("--model", type=str, required=False)
    parser.add_argument("--api-key", type=str, required=False)
    args = parser.parse_args()

    model = args.model
    if not model:
        model = input("Enter a model name for Litellm: ")

    api_key = args.api_key
    if not api_key:
        api_key = input("Enter an API key for Litellm: ")

    asyncio.run(main(model, api_key))