BUG FIX: Pydantic conversion logic for structured outputs is broken for models containing dictionaries

[dbczumar]

• I understand that this repository is auto-generated and my pull request may not be merged

Changes being requested

There's a bug in OpenAI's python client logic for translating pydantic models with dictionaries into structured outputs JSON schema definitions: dictionaries are always required to be empty in the resulting JSON schema, rendering the dictionary outputs significantly less useful since the LLM is never allowed to populate them

This PR fixes the issue and introduces test coverage.

Additional context & links

Fixes #2004

[RobertCraigie]

Thanks for the PR @dbczumar but as far as I'm aware additionalProperties: false is still required for strict mode. Did you try running this against the API?

[dbczumar]

Thanks for the PR @dbczumar but as far as I'm aware additionalProperties: false is still required for strict mode. Did you try running this against the API?

Thanks @RobertCraigie! It appears that additionalProperties: false is not required, based on testing. The current behavior of injecting additionalProperties: false for pydantic models with dictionaries is also incorrect, as discussed in #2004 (making sure we're on the same page there).

Interestingly, I did find that passing a pydantic model with a dictionary throws an error for an entirely different reason, which seems to be a separate OpenAI bug:

import json
from typing import Any, Dict

import pydantic

from openai.lib._pydantic import to_strict_json_schema

class GenerateADictionary(pydantic.BaseModel):
    resulting_dict: Dict[str, Any] = pydantic.Field(description="The dictionary to generate")

print(json.dumps(to_strict_json_schema(GenerateADictionary), indent=4))

messages = [
    {
        "role": "user",
        "content": "Call "
    }
]

response = openai.beta.chat.completions.parse(
    model="gpt-4o",
    messages=messages,
    response_format=GenerateADictionary,
)
print(json.dumps(response.choices[0].message.model_dump()['content'], indent=2))

{
    "properties": {
        "resulting_dict": {
            "description": "The dictionary to generate",
            "title": "Resulting Dict",
            "type": "object",
            "additionalProperties": false
        }
    },
    "required": [
        "resulting_dict"
    ],
    "title": "GenerateADictionary",
    "type": "object",
    "additionalProperties": false
}
Traceback (most recent call last):
  File "/Users/corey.zumar/mlexamples/dspytest/structured/oai_pyd.py", line 63, in <module>
    response = openai.beta.chat.completions.parse(
  File "/Users/corey.zumar/openai-python/src/openai/resources/beta/chat/completions.py", line 160, in parse
    return self._post(
  File "/Users/corey.zumar/openai-python/src/openai/_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/Users/corey.zumar/openai-python/src/openai/_base_client.py", line 960, in request
    return self._request(
  File "/Users/corey.zumar/openai-python/src/openai/_base_client.py", line 1064, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid schema for response_format 'GenerateADictionary': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'resulting_dict' supplied.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}

=> At a glance, it appears that the OpenAI backend may be handling required incorrectly. The schema printed above indeed seems to be a valid JSON schema.

The current PR doesn't help with this second issue:

$ python oai_pyd.py
{
    "properties": {
        "resulting_dict": {
            "description": "The dictionary to generate",
            "title": "Resulting Dict",
            "type": "object"
        }
    },
    "required": [
        "resulting_dict"
    ],
    "title": "GenerateADictionary",
    "type": "object",
    "additionalProperties": false
}
Traceback (most recent call last):
  File "/Users/corey.zumar/mlexamples/dspytest/structured/oai_pyd.py", line 63, in <module>
    response = openai.beta.chat.completions.parse(
  File "/Users/corey.zumar/openai-python/src/openai/resources/beta/chat/completions.py", line 160, in parse
    return self._post(
  File "/Users/corey.zumar/openai-python/src/openai/_base_client.py", line 1283, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/Users/corey.zumar/openai-python/src/openai/_base_client.py", line 960, in request
    return self._request(
  File "/Users/corey.zumar/openai-python/src/openai/_base_client.py", line 1064, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid schema for response_format 'GenerateADictionary': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'resulting_dict' supplied.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}

However, modifying the PR locally to remove required resolves the second issue and proves the additionalProperties: false is not required:

 $ python oai_pyd.py
{
    "properties": {
        "resulting_dict": {
            "description": "The dictionary to generate",
            "title": "Resulting Dict",
            "type": "object"
        }
    },
    "title": "GenerateADictionary",
    "type": "object",
    "additionalProperties": false
}
"{\"resulting_dict\":{\"title\":\"Incoming Phone Call\",\"description\":\"Information and options available when receiving a phone call\",\"caller_information\":{\"name\":\"+1 (415) 222-4444\",\"location\":\"San Francisco, CA, USA\"},\"call_details\":{\"ringtone\":\"Default Melody\",\"status\":\"Ringing\",\"time\":\"10:45 AM\"},\"options_available\":{\"answer_call\":\"Tap green button to answer\",\"decline_call\":\"Tap red button to decline\",\"send_text_message\":\"Tap message icon to send a quick reply\",\"set_reminder\":\"Tap to remind to call back later\"}}}"

@RobertCraigie In summary, I think there are two issues here:

1. OpenAI client injects additionalProperties: false for pydantic models containing dictionary fields, thereby forcing the LLM to never populate the dictionary fields & largely defeating the purpose of these fields in most cases

2. The OpenAI backend rejects requests with objects that don't contain properties (such as dictionaries) due to what seems to be a bug in the OpenAI backend's handling of the required field. (Other LM services don't appear to impose this restriction). I hadn't observed this before because I was testing with a different service / provider (Databricks).

Happy to file a separate issue for (2). Let me know what your thoughts are here.

[dbczumar]

@RobertCraigie Any updates / additional thoughts here?

[RobertCraigie]

At a glance, it appears that the OpenAI backend may be handling required incorrectly. The schema printed above indeed seems to be a valid JSON schema.

This part isn't a bug fwiw, the API only supports a specific subset of JSON schemas, so it is expected that valid JSON schemas cannot be used directly with strict structured outputs.

However, modifying the PR locally to remove required resolves the second issue and proves the additionalProperties: false is not required:

I was not aware that could work, let me get back to you.

[dbczumar]

@RobertCraigie the backend behavior is a bug because the error message indicates that the schema is missing a property in “required”. However, the property cited as missing in the error message is clearly contained in the payload’s “required” field. At minimum, the error message is incorrect or misleading. I suspect there shouldn’t be an error at all.

thanks for looking into this! :)

[dbczumar]

@RobertCraigie Any updates or additional thoughts here?

[joshuafcole]

Is there a known workaround for this? If not, is there any way I can help get this across the line? I may be missing something, but afaict structured output is just unusable with openai right now for non-trivial schemas.

[marcuswood-oai]

Thank you for the contribution.

We’re closing this based on the explanation and usage guidance in the closure comment on #2004.