How to have an Assistant returning JSON and using File Search?

API
1

I want to build an assistant that supports streaming, file search and responds in JSON format… but I am falling into a huge rabbit hole here.

(please see Colab Notebook for the full code, will past snippets here only h++ps://colab.research.google.com/drive/1ZApiuBIGiDmwDPC56B85tuXp_XBup9Z0?usp=sharing )

This is basically how I create the run:

prompt = f"The first criteria is `Price`."

thread = client.beta.threads.create()

thread_message = client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content=prompt
)

run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id,
    stream=True,
    # response_format="auto",
    # response_format=response_format,
    # response_format={ "type": "json_object" },
    response_format=response_format,
    tool_choice={"type": "file_search"},
    include=["step_details.tool_calls[*].file_search.results[*].content"], 
)

for event in run:
    event_type = getattr(event, "event", None)

    if event_type == "thread.message.created":
        print(f"[Message Created] Message ID: {event.data.id}")
        pass

    elif event_type == "thread.message.in_progress":
        print(f"[Message In Progress] Message ID: {event.data.id}")
        pass

    elif event_type == "thread.message.delta":
        delta_content = event.data.delta.content
        for block in delta_content:
            if block.type == "text":
                print(block.text.value)
                
    elif event_type == "thread.message.completed":
        print("\n[Message Completed]")
        print(event.data.content[0].text.value)
        print(json.loads(event.data.content[0].text.value.replace("'", '"')))

    else:
        print(event)
        pass

First approach was to use response_format=“auto”:

This is the file I want to search trough:

[
    {
        "vehicle_name": "The Explorer",
        "image_path": "shoe_data/explorer.jpg",
        "description": "The Explorer shoes are designed for those who seek adventure and durability. With a rugged exterior and comfortable interior, these shoes are perfect for hiking, trekking, and exploring the great outdoors. The high-quality materials ensure long-lasting wear, while the stylish design keeps you looking good on any trail.",
        "tech_specs_path": "shoe_data/explorer.pdf",
        "description_path": "shoe_data/explorer.txt"
    },
...
]

I get three different results (like randomly):

  1. The API responds with JSON hidden in markdown, like that:
    (adding plenks to make it renderable)
` ` `json
{...}
` ` `
  1. The API responds with no markdown but actual, but invalid JSON, like adding to much curly brackets
{foo: bar}}
  1. The API responds with no markdown but valid JSON, which is the expected result, apparently.

As this is not really helpful, I thought that I maybe can ask the API to respond in JSON-Format. So I tried:

    ... 
    response_format={ "type": "json_object" }
    ...

At least this returns JSON without the markdown wrapper, but still it is not returning valid JSON in a couple of tests - which is not robust, of course.

Next option: JSON schema. I used the Schema Generator from the playground.
However: This approach “always” (I ran 5 tests) results in a time out after roughly 10 seconds. Verbose output shows the agent is processing those steps:

ThreadRunCreated 
ThreadRunQueued 
ThreadRunInProgress 
ThreadRunStepCreated
ThreadRunStepInProgress
ThreadRunStepDelta
ReadTimeout: The read operation timed out

The playground is a little confusing:

  • when using the playground, the assistant does not support both, file search and response format. Though I can tell the assistant to respond with a JSON object or schema, but when I then choose File Search for the actual run, I get:

Error streaming run: Invalid tool_choice: FILE_SEARCH. You must also pass this tool in the ‘tools’ list on the Run.

  • the other way around works: activate file search for the assistant but response format “text” (no auto like in the docs available?)

What really bugs me, besides the fragility of the API, is that the Playground seems to work properly all the time when I choose response-format “auto” - which is the simplest approach. And I have the exact same settings in my Notebook, still it’s not running through.

I am trying to get my head around this for at least two days now. I tried dozens of iterations like, it’s always a very fragile So, any help is highly apreciated!