OpenRouter API leads to pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation. Deeper dive shown: "Plain text responses are not permitted, please call one of the functions instead"

[HomenShum]

Title: OpenRouter API Not Providing Structured Response - Exceeded Maximum Retries for Result Validation

---

Description:

When using the OpenRouter API with meta-llama/llama-3.3-70b-instruct, I encounter an issue where the structured response is not returned, resulting in an exception after exceeding the maximum number of retries. However, the same code works as expected when using the openai:gpt-4o-mini model.

---

Steps to Reproduce:

1. Setup Code:

   import asyncio
   import json
   from pydantic import BaseModel
   from some_module import (
       search_company_summary,
       OpenAIModel,
       Agent,
       OPENROUTER_API_KEY,
   )
   
   class CityLocation(BaseModel):
       city: str
       country: str
   
   async def main():
       company_name_to_search = "20n Bio"
       # Use test data
       search_results_with_verification = await search_company_summary(
           company_name=company_name_to_search, use_test_data=True
       )
       print("\n--- Final Results with Verification ---")
       print(json.dumps(search_results_with_verification, indent=2))
   
       # Using OpenRouter API model
       openrouter_model = OpenAIModel(
           'meta-llama/llama-3.3-70b-instruct',
           base_url='https://openrouter.ai/api/v1',
           api_key=OPENROUTER_API_KEY,
       )
       agent = Agent(openrouter_model, result_type=CityLocation, result_tool_name='city_location')
   
       # Using GPT 4o MINI model
       agent = Agent('openai:gpt-4o-mini', result_type=CityLocation, result_tool_name='city_location')
       result = await agent.run('Where the olympics held in 2012?')
       print(result.data)
       # Expected output:
       #   city='London' country='United Kingdom'
   
       print(result.cost())
   
   if __name__ == "__main__":
       asyncio.run(main())

2. Run the script with OpenRouter API:

   (parsely_v012325) PS D:\VSCode Projects\parsely_Jan25> & "d:/VSCode Projects/parsely_Jan25/parsely_v012325/Scripts/python.exe" "d:/VSCode Projects/parsely_Jan25/test.py"
   d:\VSCode Projects\parsely_Jan25\test.py:375: LogfireNotConfiguredWarning: No logs or spans will be created until logfire.configure() has been called. Set the environment variable LOGFIRE_IGNORE_NO_CONFIG=1 or add ignore_no_config=true in pyproject.toml to suppress this warning.
   result = await agent.run('Where the olympics held in 2012?')
   INFO:httpx:HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
   INFO:httpx:HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
   Traceback (most recent call last):
     File "d:\VSCode Projects\parsely_Jan25\test.py", line 382, in 
       asyncio.run(main())
     File "C:\Users\hshum\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
       return runner.run(main)
                ^^^^^^^^^^^^^^^
     File "C:\Users\hshum\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
       return self._loop.run_until_complete(task)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "C:\Users\hshum\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
       return future.result()
                ^^^^^^^^^^^^^^
     File "d:\VSCode Projects\parsely_Jan25\test.py", line 375, in main
       result = await agent.run('Where the olympics held in 2012?')
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "D:\VSCode Projects\parsely_Jan25\parsely_v012325\Lib\site-packages\pydantic_ai\agent.py", line 311, in run
       final_result, tool_responses = await self._handle_model_response(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "D:\VSCode Projects\parsely_Jan25\parsely_v012325\Lib\site-packages\pydantic_ai\agent.py", line 1108, in _handle_model_response
       return await self._handle_text_response(text, run_context, result_schema)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "D:\VSCode Projects\parsely_Jan25\parsely_v012325\Lib\site-packages\pydantic_ai\agent.py", line 1126, in _handle_text_response
       self._incr_result_retry(run_context)
     File "D:\VSCode Projects\parsely_Jan25\parsely_v012325\Lib\site-packages\pydantic_ai\agent.py", line 1300, in _incr_result_retry
       raise exceptions.UnexpectedModelBehavior(
   pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation
   (parsely_v012325) PS D:\VSCode Projects\parsely_Jan25>
   

3. Run the script with GPT 4o MINI (Successful Case):

   (parsely_v012325) PS D:\VSCode Projects\parsely_Jan25> & "d:/VSCode Projects/parsely_Jan25/parsely_v012325/Scripts/python.exe" "d:/VSCode Projects/parsely_Jan25/test.py"
   d:\VSCode Projects\parsely_Jan25\test.py:375: LogfireNotConfiguredWarning: No logs or spans will be created until logfire.configure() has been called. Set the environment variable LOGFIRE_IGNORE_NO_CONFIG=1 or add ignore_no_config=true in pyproject.toml to suppress this warning.
   result = await agent.run('Where the olympics held in 2012?')
   INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
   city='London' country='United Kingdom'
   

---

Additional Context:

The issue seems to be related to the way the OpenRouter API handles structured responses. The error message indicates that the model behavior is unexpected, and the code exceeds the maximum retries for result validation:

raise exceptions.UnexpectedModelBehavior(
    pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation

I suspect that the OpenRouter API does not allow plain text responses and expects a structured function call response. Meanwhile, the GPT 4o MINI model handles the structured response correctly.

---

Core Function Example (Related to Entity Description):

async def process_entity_description(content_text: str) -> EntityAndDescription:
    """
    Processes the input content text to extract entity information and break it into sections.

    Args:
        content_text: The input text content.

    Returns:
        An EntityAndDescription object containing entity name, overall description, and section breakdowns.
    """
    try:
        with capture_run_messages() as messages:
            entity_and_description_agent_result = await entity_and_description_agent.run(
                user_prompt=(
                    "Please break down the following text into smaller, manageable sections, "
                    "identify the key points, and provide a clear, decontextualized description for each section:\n\n"
                    f"{content_text}"
                )
            )
            
            return entity_and_description_agent_result.data

    except UnexpectedModelBehavior as e:
        logger.error(f"UnexpectedModelBehavior in process_entity_description: {e}")
        logger.error(f"Cause: {repr(e.__cause__)}")
        logger.error(f"Messages: {messages}")
        raise e
    except Exception as e:
        logger.error(f"Error in process_entity_description: {e}")
        raise e

---

Expected Behavior:

• When using the OpenRouter API, the model should return a structured response without exceeding the retry limit.
• The agent should successfully validate the result and output the expected structured data.

Actual Behavior:

• The OpenRouter API returns a plain text response, leading to a validation failure after one retry, and the following error is raised:

  pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation
  

• In contrast, using the GPT 4o MINI model produces the expected output.

---

Environment:

• Python Version: 3.11 (as per traceback)
• Relevant Libraries/Packages: pydantic_ai, httpx
• APIs: OpenRouter API (https://openrouter.ai/api/v1), OpenAI API (for GPT 4o MINI)

---

Additional Notes:

• The warning about LogfireNotConfiguredWarning is noted but does not appear to be directly related to the structured response issue.
• The issue was identified by closely inspecting the captured run messages and traceback.

---

Request:

Could you please investigate why the OpenRouter API is not returning the structured response and causing a result validation failure? Any insights into how to configure the API or modify the code to handle this behavior would be appreciated.

---

[samuelcolvin]

Please format the could and terminal output as markdown properly.

This is unreadable.

[HomenShum]

Please format the could and terminal output as markdown properly.

This is unreadable.

Thank you for the feedback, updated with edit.

[sydney-runkle]

Hmm, this just seems like a model limitation to me...

[jonchun]

Agreed with @sydney-runkle - Llama 70b can struggle to properly call tools sometimes. Can you try posting some DEBUG level output so we can see the actual HTTP requests/responses from httpx? Make sure you strip out any sensitive data -- can't remember if it's included off the top of my head.

[github-actions]

This issue is stale, and will be closed in 3 days if no reply is received.

[mvaldi]

Hi! I have the same error but for every model in OpenRouter

For me, OpenRouter is just OpenAI wrapper, in all other models it throws an error, it can be the one present in this issue, or can't handle tools

I let some code to test it

This is a dummy code to reproduce those errors, and take a look to Enum variable, because the models of OpenRouter can't handle correctly with this param, only openai can

from enum import Enum
from typing import Literal

from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.result import RunResult

agent_name: Literal[
    "google/gemini-2.0-flash-001",  # Structured error
    "qwen/qwen-plus",  # Structured error
    "meta-llama/llama-3.3-70b-instruct",  # Structured error
    "microsoft/phi-4",  # Don't support tool
    "openai/gpt-4o-mini",  # the only one that works, and every openai model
] = "google/gemini-2.0-flash-001"

model = OpenAIModel(
    model_name=agent_name,
    base_url="https://openrouter.ai/api/v1",
    api_key="API_KEY",
)

# Set user category by database
# for this example we will use some classes from imagenet (100-110)
user_category = [
    "black swan, Cygnus atratus",
    "tusker",
    "echidna, spiny anteater, anteater",
    "platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhyhus anatinus",
    "wallaby, brush kangaroo",
    "koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus",
    "wombat",
    "jellyfish",
    "sea anemone, anemone",
    "brain coral",
    "flatworm, platyhelminth",
]

# convert list to Enum to use in pydantic
user_category_to_pydantic = Enum(
    "UserCategory",
    {category: category for category in user_category},
)


class ResultData(BaseModel):
    category: user_category_to_pydantic


agent = Agent(
    model,
    system_prompt="You must to classify the category of the following text.",
    deps_type=None,
    result_type=ResultData,
    model_settings={"temperature": 0},
    retries=3,
)

# throw Exceeded Maximum Retries for Result Validation Error
answer: RunResult[ResultData] = agent.run_sync(
    "Day 2 on my experiments: I saw a jellyfish in the sea, it was a beautiful day.",
)


def upload_to_db(category: user_category_to_pydantic): ...


upload_to_db(answer.data.category)

Output `google/gemini-2.0-flash-001`

The text in `_agent_graph.py:351` is

first attempt

Personal Blog/Diary Entry

third attempt

I apologize again for the continued errors. I am still struggling to adhere to the "no plain text" rule. I will try a different approach to simulate the function call and return.

``
SYSTEM: Calling function 'classify_text' with input: "Day 2 on my experiments: I saw a jellyfish in the sea, it was a beautiful day."
``

``
SYSTEM: Function 'classify_text' returned: "Personal Blog / Diary Entry"
``

This simulates the system calling the function and then receiving the result. Is this a better representation?

Output `meta-llama/llama-3.3-70b-instruct`

The text in `_agent_graph.py:351` is

<function=final_result>{"category": "jellyfish"}</function>

Output `qwen/qwen-plus`

The text in `_agent_graph.py:351` is

_function\n✿ARGS✿: {"category": "jellyfish"}

What seems strange to me most is that Gemini 2.0 Flash, works with vertex-ai

from enum import Enum
from typing import Literal

from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.models.vertexai import VertexAIModel
from pydantic_ai.result import RunResult

model = VertexAIModel(
    model_name="gemini-2.0-flash",
    project_id="project_id",
)

# Set user category by database
# for this example we will use some classes from imagenet (100-110)
user_category = [
    "black swan, Cygnus atratus",
    "tusker",
    "echidna, spiny anteater, anteater",
    "platypus, duckbill, duckbilled platypus, duck-billed platypus, Ornithorhyhus anatinus',",
    "wallaby, brush kangaroo",
    "koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus",
    "wombat",
    "jellyfish",
    "sea anemone, anemone",
    "brain coral",
    "flatworm, platyhelminth",
]

# convert list to Enum to use in pydantic
user_category_to_pydantic = Enum(
    "UserCategory",
    {category: category for category in user_category},
)


class ResultData(BaseModel):
    category: user_category_to_pydantic


agent = Agent(
    model,
    system_prompt="You must classify the category of the following text.",
    deps_type=None,
    result_type=ResultData,
    model_settings={"temperature": 0},
    retries=3,
)

answer: RunResult[ResultData] = agent.run_sync(
    "Day 2 on my experiments: I saw a jellyfish in the sea, it was a beautiful day.",
)


def upload_to_db(category: user_category_to_pydantic): ...


upload_to_db(answer.data.category)

I don't know why Gemini 2.0 works with vertex-ai, but crashed with OpenRouter

[github-actions]

This issue is stale, and will be closed in 3 days if no reply is received.

[hifsa-rehman]

why we are not able to build ai agents with tools using free tier models using openrouter

[TheRockPusher]

Gemini flash 1.5 and Claude also work (if you add a ton of instructions to just tool call and not write text in the prompt), albeit not all the time, Gemini 2.0 is almost impossible to make work, only OpenAI models work consistently, even weaker ones. Models just don't like to execute the response tool call that pydanticai is expecting, especially through OpenRouter for some reason.

Overall for everything that is not OpenAI, using the ResultType validation has been kinda useless on my testing, you get way more consistent results by asking the model to return Json and after that parse it and verify it with a model.

But I guess structured outputs that is in the roadmap would help with this issue.

[github-actions]

This issue is stale, and will be closed in 3 days if no reply is received.