Summary: gpt-4o-mini suddenly started producing invalid Enum values, leading to Pydantic ValidationErrors in the OpenAI client.
I’m using gpt-4o-mini-2024-07-18 with a JSON schema to get responses in form of Pydantic models. The Pydantic model uses a field of the Enum type, like this:
class DamageLevelLLM(str, Enum):
none = "none"
minor = "minor - up to tens of thousands USD"
moderate = "moderate - hundred thousands USD"
major = "major - millions USD"
catastrophic = "catastrophic - large tens of millions or more"
class EventUpdate(BaseModel):
title: str = Field(title="Event Title", description="Title of the event.")
damages: DamageLevelLLM = Field(title="Damages", description="Your estimation of the level of property damages caused by the event. Do not include casualties or injuries.")
I’m using the following code to query the model:
client = OpenAI(api_key=config.openai.api_key)
response = client.beta.chat.completions.parse(
model=config.openai.model,
messages=[
{
"role": "system",
"content": [
{
"text": system_instruction,
"type": "text"
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": user_prompt
}
]
}
],
temperature=temperature,
max_tokens=max_tokens,
top_p=top_p,
frequency_penalty=frequency_penalty,
presence_penalty=presence_penalty,
response_format=EventUpdate
)
result = response.choices[0].message.parsed
Note: All the parameters are in their default values.
Everything was working fine for several weeks. Now today something changed and I’m getting pydantic_core._pydantic_core.ValidationError: 1 validation error.
The code crashes on the client.beta.chat.completions.parse() call, with the following stack trace:
File "/Users/michal/project-name/llm/helper.py", line 18, in query_llm
response = client.beta.chat.completions.parse(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/resources/beta/chat/completions.py", line 154, in parse
return self._post(
^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 1277, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 954, in request
return self._request(
^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 1060, in _request
return self._process_response(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_base_client.py", line 1159, in _process_response
return api_response.parse()
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_response.py", line 319, in parse
parsed = self._options.post_parser(parsed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/resources/beta/chat/completions.py", line 148, in parser
return _parse_chat_completion(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/lib/_parsing/_completions.py", line 110, in parse_chat_completion
"parsed": maybe_parse_content(
^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/lib/_parsing/_completions.py", line 161, in maybe_parse_content
return _parse_content(response_format, message.content)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/lib/_parsing/_completions.py", line 221, in _parse_content
return cast(ResponseFormatT, model_parse_json(response_format, content))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/openai/_compat.py", line 166, in model_parse_json
return model.model_validate_json(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/project-name/lib/python3.11/site-packages/pydantic/main.py", line 625, in model_validate_json
return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for EventUpdate
pydantic_core._pydantic_core.ValidationError: 1 validation error for EventUpdate
damages
Input should be 'none', 'minor - up to tens of thousands USD', 'moderate - hundred thousands USD', 'major - millions USD' or 'catastrophic - large tens of millions or more' [type=enum, input_value='major', input_type=str]
For further information visit https://errors.pydantic.dev/2.9/v/enum
As you can see, the model produces invalid response for the field “major”, instead of properly outputing the full Enum value.
I believe there must have been a change to the model, because there was no code change on my side, nor did I update any of the packages.
Will be grateful for any pointers on how to solve this.
