OpenAI Python client fails to parse Pydantic model

API Bugs
1

Summary: gpt-4o-mini suddenly started producing invalid Enum values, leading to Pydantic ValidationErrors in the OpenAI client.

I’m using gpt-4o-mini-2024-07-18 with a JSON schema to get responses in form of Pydantic models. The Pydantic model uses a field of the Enum type, like this:


class DamageLevelLLM(str, Enum):
    none = "none"
    minor = "minor - up to tens of thousands USD"
    moderate = "moderate - hundred thousands USD"
    major = "major - millions USD"
    catastrophic = "catastrophic - large tens of millions or more"

class EventUpdate(BaseModel):

    title: str = Field(title="Event Title", description="Title of the event.")
    damages: DamageLevelLLM = Field(title="Damages", description="Your estimation of the level of property damages caused by the event. Do not include casualties or injuries.")

I’m using the following code to query the model:

    client = OpenAI(api_key=config.openai.api_key)

    response = client.beta.chat.completions.parse(
        model=config.openai.model,
        messages=[
            {
                "role": "system",
                "content": [
                    {
                        "text": system_instruction,
                        "type": "text"
                    }
                ]
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": user_prompt
                    }
                ]
            }
        ],
        temperature=temperature,
        max_tokens=max_tokens,
        top_p=top_p,
        frequency_penalty=frequency_penalty,
        presence_penalty=presence_penalty,
        response_format=EventUpdate
    )

    result = response.choices[0].message.parsed

Note: All the parameters are in their default values.

Everything was working fine for several weeks. Now today something changed and I’m getting pydantic_core._pydantic_core.ValidationError: 1 validation error.

The code crashes on the client.beta.chat.completions.parse() call, with the following stack trace:

  File "/Users/michal/project-name/llm/helper.py", line 18, in query_llm
    response = client.beta.chat.completions.parse(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/resources/beta/chat/completions.py", line 154, in parse
    return self._post(
           ^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 1277, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 954, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 1060, in _request
    return self._process_response(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 1159, in _process_response
    return api_response.parse()
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_response.py", line 319, in parse
    parsed = self._options.post_parser(parsed)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/resources/beta/chat/completions.py", line 148, in parser
    return _parse_chat_completion(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/lib/_parsing/_completions.py", line 110, in parse_chat_completion
    "parsed": maybe_parse_content(
              ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/lib/_parsing/_completions.py", line 161, in maybe_parse_content
    return _parse_content(response_format, message.content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/lib/_parsing/_completions.py", line 221, in _parse_content
    return cast(ResponseFormatT, model_parse_json(response_format, content))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_compat.py", line 166, in model_parse_json
    return model.model_validate_json(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/pydantic/main.py", line 625, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for EventUpdate
pydantic_core._pydantic_core.ValidationError: 1 validation error for EventUpdate
 damages
   Input should be 'none', 'minor - up to tens of thousands USD', 'moderate - hundred thousands USD', 'major - millions USD' or 'catastrophic - large tens of millions or more' [type=enum, input_value='major', input_type=str]
        For further information visit https://errors.pydantic.dev/2.9/v/enum

As you can see, the model produces invalid response for the field “major”, instead of properly outputing the full Enum value.

I believe there must have been a change to the model, because there was no code change on my side, nor did I update any of the packages.

Will be grateful for any pointers on how to solve this.

1 Like
2

Hey @miso ! For cases like these, you may use try and except blocks. Have the inference code in try block; while catching pedantic errors have a retry mechanism that passes error, json as prompt and infer to fix it. Con: Limited by ip op token limit.

Hape this helps. Cheers :slightly_smiling_face:

3

Thank you for the hint @Munna23 . I think that makes a lot of sense, however, in this case it seems the model produces invalid enum every time, or at least almost every time :frowning:

4

model_config = ConfigDict(use_enum_values=True)

Have you tried this option in the model_config

5

Thanks for trying to help out, but there is no such option needed.

When using the response_format parameter, and when you provide the OpenAI beta “parse” method of its Python SDK library a BaseModel from Pydantic, this always sends a “strict” schema. The AI should be able to produce nothing else but enums given when it is at a key that has an enum spec, by using Literal.

This is either a fluke of the API, long since resolved, or a problem with the Pydantic model and ambiguity between the schema and the validation. One can grab the API JSON request that is being made by the Python client by giving it a different placeholder transport instead of httpx, and see what is actually being sent in the underlying JSON requests.