Streaming is now available in the Assistants API!

API
, , ,
44

Hi hq1 - take a look at the OpenAI Assistant Starter Kit which illustrates how to use Assistant streaming with a NextJS app.

Live App Here: https://openai-assistant-starter-kit.vercel.app/
Blog Walkthrough Here: Use the OpenAI Assistant Starter Kit to Quickly Build New OpenAI Apps - OpenAI Blog - Stephen Walther on OpenAI
Full Source Code Here: GitHub - Superexpert/openai-assistant-starter-kit: Starter Kit for creating an OpenAI Assistant web application using NextJS + ReactJS + TypeScript

The app streams from the server and then parses out the Server Side Events in a ReactJS component. I’m not using any special libraries. Feedback welcomed and appreciated!

4 Likes
45

Take a look to this:

46

to update - i pivoted to using the vercel ai sdk and could get the streaming working brilliantly. I am still curious why I could not get the websocket to return the stream, but have moved to the vercel way of doing it (for now).

I would love to hear an update, if the websocket streaming is working with a frontned… thx

47

Hi - I will go and check my python code again, as would really like to get this working, and really appreciate the pointers. I did pivot to using the vercel ai sdk and got a working streaming in < 10 minutes.

I will give your suggestions a try and come back with an update.

thanks again!
hq

1 Like
48

Hi everyone, does anyone have an idea if streaming the message object during an assistant call consumes the RPM limit?

  • 60 req/min limit at the user account level.
49

i’m also struggling with submit_tools_output_stream.

50

How to save this messages history. I was using run.id to retrieve, but now I can’t find out id for each run.

if prompt := st.chat_input("How can I help you?"):
    # Add user message to the state and display on the screen
    st.session_state.messages.append({"role": "user", "content": prompt})
    if prompt:
        run=client.beta.threads.runs.create_and_stream(
                thread_id=thread_id,
                assistant_id=assistant_id,
                model="gpt-4-turbo-preview",
            ) 
        with run as stream:
                with st.chat_message("assistant"):
                    response = st.write_stream(stream.text_deltas)
                    stream.until_done()
    with st.chat_message("user"):
        st.markdown(prompt)
53

You have to uninstall and reinstall the latest version of openai SDK

54

@shahir1 I updated my library to 1.27.0 and it still has this import issue

55

I’m trying to build a streaming app with openAI assistants where there are two assistants that communicate with each other. I want to stream the interaction between the two back to the frontend, without the end user having to do anything. Do you have any examples of that?

Thanks!

56

Hi smiith – you should be able to adapt the OpenAI Assistant Starter Kit code to run threads associated with two different assistants by modifying the server-side code in the route.ts file.

  1. You’ll need to setup two assistants at the OpenAI playground: https://platform.openai.com/playground?mode=assistant

  2. Next, you can build two different threads:

   // add new message to thread
    await openai.beta.threads.messages.create(
        newMessage.threadId,
        {
            role: "user",
            content: newMessage.content
        }
    );
  1. Next, run both of the threads using a particular assistant by using the Assistant Id (you can get the Assistant Ids from the OpenAI playground).
    // create a run
    const stream = await openai.beta.threads.runs.create(
        newMessage.threadId, 
        {assistant_id: newMessage.assistantId, stream:true}
    );

You can pass the responses from the two different assistants back and forth by adding the response messages to each thread.

Hope this helps! And sounds like an intriguing project!

Stephen

2 Likes
57

I was able to get it to work when I changed my python runtime from 3.11 to 3.8

58

Still waiting on documentation of the AssistantEventHandler. I see various examples floating around, some of which override certain events, while others do not. It’s unclear when that is necessary, or even what a complete list of the events are. GPT itself doesn’t know either.

59 
with client.beta.threads.runs.stream(
        thread_id=thread_id,
        assistant_id=ASS_ID,
    ) as stream:

With the above, I get this error: AttributeError: 'Runs' object has no attribute 'stream', meanwhile it is there.

When I decide to use the AssistantEventHandler, I get this AttributeError: 'Runs' object has no attribute 'stream'.
Meanwhile, when I import AssistantEventHandler to a Jupyter notebook, it imports, but in my script, I get that error.

I’m on version 1.45.1, I downgraded to 1.42.0 but got the same errors.

60

It worked now. I think my terminal was on a different environment, which might have had a different openai version.

1 Like