Structured Output - Why isn't `dict` a supported type? (to temporarily turn off restrictions)

[bpechounov]

Is dict equivalent to Object? When I make a request with a field of type dict, I get a 400 error.

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid schema for response_format 'Invoice': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'invoice_number' supplied.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}

I am thinking that dict would simply temporarily turn off restrictions, so it should be “easy” to implement.

---

It seems that dict indeed corresponds to object.

from pydantic import BaseModel, Field
from rich import print as pprint

class Invoice(BaseModel):
    invoice_number: dict = Field(description="Unique identifier for the invoice")

pprint(Invoice.model_json_schema())


{
    'properties': {
        'invoice_number': {
            'description': 'Unique identifier for the invoice',
            'title': 'Invoice Number',
            'type': 'object'
        }
    },
    'required': ['invoice_number'],
    'title': 'Invoice',
    'type': 'object'
}

For a full stack trace:

from langchain_openai import AzureChatOpenAI

llm = AzureChatOpenAI(azure_deployment="gpt-4o", api_version="2024-08-01-preview")
llm.with_structured_output(Invoice, method="json_schema").invoke("")

---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
Cell In[2], line 4
      1 from langchain_openai import AzureChatOpenAI
      3 llm = AzureChatOpenAI(azure_deployment="gpt-4o", api_version="2024-08-01-preview")
----> 4 llm.with_structured_output(Invoice, method="json_schema").invoke("")

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_core\runnables\base.py:3027, in RunnableSequence.invoke(self, input, config, **kwargs)
   3025 context.run(_set_config_context, config)
   3026 if i == 0:
-> 3027     input = context.run(step.invoke, input, config, **kwargs)
   3028 else:
   3029     input = context.run(step.invoke, input, config)

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_core\runnables\base.py:5365, in RunnableBindingBase.invoke(self, input, config, **kwargs)
   5359 def invoke(
   5360     self,
   5361     input: Input,
   5362     config: Optional[RunnableConfig] = None,
   5363     **kwargs: Optional[Any],
   5364 ) -> Output:
-> 5365     return self.bound.invoke(
   5366         input,
   5367         self._merge_configs(config),
   5368         **{**self.kwargs, **kwargs},
   5369     )

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_core\language_models\chat_models.py:307, in BaseChatModel.invoke(self, input, config, stop, **kwargs)
    296 def invoke(
    297     self,
    298     input: LanguageModelInput,
   (...)
    302     **kwargs: Any,
    303 ) -> BaseMessage:
    304     config = ensure_config(config)
    305     return cast(
    306         ChatGeneration,
--> 307         self.generate_prompt(
    308             [self._convert_input(input)],
    309             stop=stop,
    310             callbacks=config.get("callbacks"),
    311             tags=config.get("tags"),
    312             metadata=config.get("metadata"),
    313             run_name=config.get("run_name"),
    314             run_id=config.pop("run_id", None),
    315             **kwargs,
    316         ).generations[0][0],
    317     ).message

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_core\language_models\chat_models.py:843, in BaseChatModel.generate_prompt(self, prompts, stop, callbacks, **kwargs)
    835 def generate_prompt(
    836     self,
    837     prompts: list[PromptValue],
   (...)
    840     **kwargs: Any,
    841 ) -> LLMResult:
    842     prompt_messages = [p.to_messages() for p in prompts]
--> 843     return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_core\language_models\chat_models.py:683, in BaseChatModel.generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs)
    680 for i, m in enumerate(messages):
    681     try:
    682         results.append(
--> 683             self._generate_with_cache(
    684                 m,
    685                 stop=stop,
    686                 run_manager=run_managers[i] if run_managers else None,
    687                 **kwargs,
    688             )
    689         )
    690     except BaseException as e:
    691         if run_managers:

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_core\language_models\chat_models.py:908, in BaseChatModel._generate_with_cache(self, messages, stop, run_manager, **kwargs)
    906 else:
    907     if inspect.signature(self._generate).parameters.get("run_manager"):
--> 908         result = self._generate(
    909             messages, stop=stop, run_manager=run_manager, **kwargs
    910         )
    911     else:
    912         result = self._generate(messages, stop=stop, **kwargs)

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\langchain_openai\chat_models\base.py:683, in BaseChatOpenAI._generate(self, messages, stop, run_manager, **kwargs)
    678         warnings.warn(
    679             "Cannot currently include response headers when response_format is "
    680             "specified."
    681         )
    682     payload.pop("stream")
--> 683     response = self.root_client.beta.chat.completions.parse(**payload)
    684 elif self.include_response_headers:
    685     raw_response = self.client.with_raw_response.create(**payload)

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\openai\resources\beta\chat\completions.py:156, in Completions.parse(self, messages, model, audio, response_format, frequency_penalty, function_call, functions, logit_bias, logprobs, max_completion_tokens, max_tokens, metadata, modalities, n, parallel_tool_calls, prediction, presence_penalty, seed, service_tier, stop, store, stream_options, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
    149 def parser(raw_completion: ChatCompletion) -> ParsedChatCompletion[ResponseFormatT]:
    150     return _parse_chat_completion(
    151         response_format=response_format,
    152         chat_completion=raw_completion,
    153         input_tools=tools,
    154     )
--> 156 return self._post(
    157     "/chat/completions",
    158     body=maybe_transform(
    159         {
    160             "messages": messages,
    161             "model": model,
    162             "audio": audio,
    163             "frequency_penalty": frequency_penalty,
    164             "function_call": function_call,
    165             "functions": functions,
    166             "logit_bias": logit_bias,
    167             "logprobs": logprobs,
    168             "max_completion_tokens": max_completion_tokens,
    169             "max_tokens": max_tokens,
    170             "metadata": metadata,
    171             "modalities": modalities,
    172             "n": n,
    173             "parallel_tool_calls": parallel_tool_calls,
    174             "prediction": prediction,
    175             "presence_penalty": presence_penalty,
    176             "response_format": _type_to_response_format(response_format),
    177             "seed": seed,
    178             "service_tier": service_tier,
    179             "stop": stop,
    180             "store": store,
    181             "stream": False,
    182             "stream_options": stream_options,
    183             "temperature": temperature,
    184             "tool_choice": tool_choice,
    185             "tools": tools,
    186             "top_logprobs": top_logprobs,
    187             "top_p": top_p,
    188             "user": user,
    189         },
    190         completion_create_params.CompletionCreateParams,
    191     ),
    192     options=make_request_options(
    193         extra_headers=extra_headers,
    194         extra_query=extra_query,
    195         extra_body=extra_body,
    196         timeout=timeout,
    197         post_parser=parser,
    198     ),
    199     # we turn the `ChatCompletion` instance into a `ParsedChatCompletion`
    200     # in the `parser` function above
    201     cast_to=cast(Type[ParsedChatCompletion[ResponseFormatT]], ChatCompletion),
    202     stream=False,
    203 )

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\openai\_base_client.py:1278, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
   1264 def post(
   1265     self,
   1266     path: str,
   (...)
   1273     stream_cls: type[_StreamT] | None = None,
   1274 ) -> ResponseT | _StreamT:
   1275     opts = FinalRequestOptions.construct(
   1276         method="post", url=path, json_data=body, files=to_httpx_files(files), **options
   1277     )
-> 1278     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\openai\_base_client.py:955, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
    952 else:
    953     retries_taken = 0
--> 955 return self._request(
    956     cast_to=cast_to,
    957     options=options,
    958     stream=stream,
    959     stream_cls=stream_cls,
    960     retries_taken=retries_taken,
    961 )

File c:\Users\BogdanPechounov\miniconda3\envs\ml-3.10\lib\site-packages\openai\_base_client.py:1059, in SyncAPIClient._request(self, cast_to, options, retries_taken, stream, stream_cls)
   1056         err.response.read()
   1058     log.debug("Re-raising status error")
-> 1059     raise self._make_status_error_from_response(err.response) from None
   1061 return self._process_response(
   1062     cast_to=cast_to,
   1063     options=options,
   (...)
   1067     retries_taken=retries_taken,
   1068 )

BadRequestError: Error code: 400 - {'error': {'message': "Invalid schema for response_format 'Invoice': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'invoice_number' supplied.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}

[aprendendo.next]

A generic dict would defeat the purpose of a structured output, because the model can’t know in advance what keys it has to return.

Instead, you declare a class with the keys (properties) you expect, and combine it like this:

class InvoiceDetails(BaseModel):
    description: str
    customer_name: str

class Invoice(BaseModel):
    invoice_number: str
    details: InvoiceDetails

print(json.dumps(Invoice.model_json_schema(), indent=4))

output

{
“$defs”: {
“InvoiceDetails”: {
“properties”: {
“description”: {
“title”: “Description”,
“type”: “string”
},
“customer_name”: {
“title”: “Customer Name”,
“type”: “string”
}
},
“required”: [
“description”,
“customer_name”
],
“title”: “InvoiceDetails”,
“type”: “object”
}
},
“properties”: {
“invoice_number”: {
“title”: “Invoice Number”,
“type”: “string”
},
“details”: {
“$ref”: “#/$defs/InvoiceDetails”
}
},
“required”: [
“invoice_number”,
“details”
],
“title”: “Invoice”,
“type”: “object”
}

Then you pass it into the API using response_format:

    response = client.beta.chat.completions.parse(
    #...other parameters
      response_format=Invoice
    )

[_j]

@bpechounov

Is dict equivalent to Object?

Essentially, yes.

The schema is defined by JSON terminology

A dictionary is a key-value container like “object”
A list is an ordered container of items like “array”
JSON stumbles a bit with “number” instead of typed int, float32, etc
Then True->true, False->false, None->null

If you use a streamable such as a dict or array to JSON, Pydantic schema as response_format (or “text”, “format” on Responses), the thinking is taken care of for you, just as if using requests.get(AI_URL, headers, json=my_dict) for RESTful calls.

**kwargs in the shown context.run() is “code” for keyword arguments. It will turn a dict into function parameters. The earlier parameters are positional.

---

In terms of “strict” schema, you must have a defined set of keys: the AI can’t fabricate its own, or write indeterminate length objects.

[havryliuk]

Yes, I believe this is the correct answer.

I had an initial structure like this:
{
“sentences”: {
“word1”: “Sentence with <word1>.”,
“word2”: “Sentence with <word2>.”
}
}

Which is definitely understandable for a human and less verbose.

And ended up with this:
{
“sentences”: [{
“word”: “word1”,
“sentence”: “Sentence with <word1>.”
},
{
“word”: “word2”,
“sentence”: “Sentence with <word2>.”
}
}
which actually is a more robust representation of a really “structured” output.