Structured Outputs with Assistants

API
1

Has anyone got the new Structured Responses working with Assistants?

I keep getting ‘Object of type ModelMetaclass is not JSON serializable’

eg.

from pydantic import BaseModel

class Insights(BaseModel):
        segments: List[str]
        garm: List[str]
        reasoning: str

run = await openai_client.beta.threads.runs.create(
        model="gpt-4o-2024-08-06",
        thread_id=thread_id,
        assistant_id='',
        temperature=0,
        response_format=Insights
    )

Am I doing something wrong? I am using the Async client too.

1 Like
2

Following the example code on the documentation page I also could not get the code to run - was getting the same exception.

Try writing it like the code below to fix it:

         response_format={
           'type': 'json_schema',
           'json_schema': 
              {
                "name":"whocares", 
                "schema": Insights.model_json_schema()
              }
         }
10 Likes
3

Thanks, they really need to add this to the Documentation.

4

Firstly, thank you for this. It really does need to be added to the documentation. The docs are not clear at all.

It should be noted that all tools must be of type function when response_format is of type json_schema.

This is/was in the documentation in some form but it is not made clear. The documentation at https://platform.openai.com/docs/guides/structured-outputs/introduction makes it sound like you can use it When using function calling or When using a json_schema response format but that does not apply to assistants.

I tried implementing json_schema with tools=[{"type": "file_search"}] and it is still not a thing. I’ve needed this feature so it’s more like chat completions for a while but right now; I’m having to convince the assistant API to respond with JSON through some kludgery instead, and then dealing with broken JSON output through exception handling.

Ugghhh.

1 Like
5

i spent so much time trying to troubleshoot this. thank you for creating this topic.

2 Likes
6

I’m dealing with the same problem - not being able to use the tools. One workaround is to do two consecutive runs for a given prompt - first one to solve the problem using whatever the tools you need to use. The second one with the structured output and the prompt “please output the results as json” - the results from the first run will be in the immediate message history so should work ok.

7

I am still having issues with setting the assistant with the structured outputs.
First, it’s unclear where we should set it(nothing was mentioned in the documentation). in the creation of the assistant/messages/threads?
secondly, I tried out the suggestion above as follows:

    class Step(BaseModel):
        explanation: str
        output: str

    
    class ReportReasoning(BaseModel):
        steps: list[Step]
        final_answer: str

        new_run = client.beta.threads.runs.create(
            thread_id=new_thread.id,
            assistant_id=assistant.id,
            response_format={
                'type': 'json_schema',
                'json_schema':
                    {
                        "name": "whocares",
                        "schema": ReportReasoning.model_json_schema()
                    }},
        )

THIS causes the next error -

openai.BadRequestError: Error code: 400 - {'error': {'message': 'Invalid tools: all tools must be of type `function` when `response_format` is of type `json_schema`.', 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}
8

The answer is in the error message. You can use any other tools than your own functions. So no code interpreter, no file search. you can set tools = in the call to create the run - this will override whatever tools you have set up on the agent. This way you can at least check, that the structured output part is working. To use tools split your workflow into two parts - create a run where you try to solve the task given by the prompt with all the tools you need. In the second run disable all the tools but set the structured output schema and in the prompts ask the agent to “please output the results as the given json” or something of the like.

1 Like
9

I’m still struggling to find a solution when using the Assistants API.

Similar to the issue raised by RvB1701 , I’m unsure where we’re supposed to specify the response type. I’m a bit confused, and I’m starting to wonder if I need to specify the response type when retrieving the messages after the run has completed.

@vismantas I don’t think we’re supposed to pass a Pydantic model directly to response_format. While you could supply the JSON schema, I’ve read that if you specify the Pydantic model, it’s supposed to return it as a Pydantic model.

@devnotdev Did you get your solution working?

I really hope to get some clarity on this because it’s crucial for what I’m working on.

1 Like
10

AFAIK Assistants API and Completions API behave slightly differently. Specifically Completions API does accept pydantic model directly, Assistants API, at least at the time of originally answering this questions were not. I’ve not checked the API for changes since, so this might be outdated.

2 Likes
11

@RvB1701

I was able to get my code to run using .model_json_schema to define the schema. Like @vismantas mentioned, it doesn’t look like the Assistant API fully supports pydantic models yet. You can still define the model with pydantic but then use .model_json_schema() to convert the pydantic model into a json schema.

The only difference I can think of might have to do with which tools you are using with your assistant. I am not using any tools right now as I was just trying to get the Structured Output to work.

I used a Jupyter notebook, and setup the client, thread, and message using the documentation. I then updated my run as follows:

from pydantic import BaseModel
from typing import List

class Message(BaseModel):
    message: str

class Messages(BaseModel):
    messages: List[Message]

run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "test_schema",
            "schema": Messages.model_json_schema()
        }
    }
)
12

Quite sloppy that they don’t mention that a pydantic model is not directly accepted as a response format in the assistants API at all in their documentation…

1 Like
13

I can’t make it work even with your solution:

class FieldRule(BaseModel):
    selector_type: str
    selectors: List[str]
    attribute: Optional[str]

class Rules(BaseModel):
    title: FieldRule
    description: Optional[FieldRule]
    link: FieldRule

class RuleSet(BaseModel):
    name: str
    rules: Rules

class ExtractionRules(BaseModel):
    rules_sets: List[RuleSet]
    has_news: bool

def extract_xpaths(html_content):
    try:
        completion = openai_client.beta.chat.completions.parse(
            model="gpt-4o-2024-08-06",
            messages=[
                {"role": "system", "content": system_instruction},
                {"role": "user", "content": html_content}
            ],
            response_format={
           'type': 'json_schema',
           'json_schema': 
              {
                "name":"ExtractionRules", 
                "schema": ExtractionRules.model_json_schema()
              }
         } 
        )
        if completion and completion.choices and len(completion.choices) > 0:
            return completion.choices[0].message.parsed
        else:
            logger.error("Invalid completion response from OpenAI")
            return None
    except Exception as e:
        logger.error(f"Error in extract_xpaths: {str(e)}")
        return None

Error:
Object of type Tag is not JSON serializable

Would appreciate any help.

14

after searching in the github repo of openai i found out that the pydantic models are not allowed to have default values and have other problems with pydantic model schema generations… my guess is that you use Optional[FieldRule] in your model.
here is the source

1 Like
15

Like @TheZeke mentioned earlier, it’s still impossible to combine file_search with JSON response functions. As a workaround until an official update is released, I created a function called ‘extract_my_data’ which allows me to access my data in a JSON schema format when triggered.

And I just ignore the output of the run.

16

What model deployment are you using and what version? I’m getting

'error': {'message': "Invalid parameter: 'response_format' of type 'json_schema' is not supported with model version `gpt-4o-08-06`.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}

with 2024-08-06 using the exact codes as you are.

17 
"response_format": {
    "type": "json_object"
},

“tool_resources”: {
“code_interpreter”: null,
“file_search”: {
“vector_store_ids”: [
“vs_QkrKgLCYpASSDF21333Z2apeo”
]
}
},
Segun tengo entendido ofrecen 2 maneras de especificar el formato de respuesta: un json_object o un json_schema que pertenece a base model creo.
Si mal no recuerdo esto tambien se puede agregar con el file_search pero agregandolo desde la pagina oficial, me funciono correctamente las 3 herramientas: function_calling , response_format y file_search.