Any updates on Assistant API Streaming?

[ahmadsameer]

Waiting for this as well…please support streaming ASAP, Thanks

[kpebedko_1]

What do we want? - Streaming in Assistants API!
When do we want it? - Now!

[szymon.kurcab]

Adding a link to other thread I’ve started on this:

Steaming in Assistant API

Hey! Does anyone know when output streaming will be available for the Assistant API - mentioned here Assistant API Limitations - will be implemented? Assistant API is great and solves lots of different problems, but w/o streaming it’s kind of unusable production-wise (especially with RAG + function calls). I know this topic was mentioned couple of times here, but no response. Maybe there as some OpenAI people here reading I’m more than happy to help with it if needed …

+1 to steaming in the Assistant API

[Dojima]

Do you guys have any alternative to Code Interpreter like the one from the Assistant API? I plan to use chat completion endpoint while waiting Assistant API streaming, but I don’t know how to simulate the code sandboxing.

[jhakulin]

Before streaming is fully supported, there is an option to check by polling (e.g. I use every 1 second) if there are new message IDs from the assistant (role) in the thread’s messages list and if there are then flush the previous messages and display all assistant messages again. This will produce streaming like effect, where assistant responses appear during the in progress state.
It depends of course on your scenario if there are those intermediate messages or not. E.g. when creating charts with code interpreter they often happen during the run.

[vinodvarma24]

Needed: Streaming responses to Assistant API

We have been waiting for this, Uses are not satisfied to wait 20-30 seconds for each response

[kashishvarshney63]

I don’t know how long we have to wait. its been so long now and so irritating

[karsten.wysk]

I would just like to voice my support to get streaming for Assistants as well

[shira.elitzur]

Is it only the lack of streaming?
Or is it also really, really slow?
Cause even when polling, I don’t get an answer sometimes for long minutes and not always because of a long answer… using functions for example.

[vonv]

I’ve noticed that the API is running slow recently. Before, it only runs slow if using any of the gpt-4 models. But in the past two days it runs slow even for the gpt-3 models, that used to run really fast.

[Bettorfaster]

Building viable chat based applications from a quality UX standpoint is simply not possible without streaming. A lack of information or roadmap transparency from openAI on the matter makes it difficult for customers like us to make informed business decisions on whether we should invest in solving problems with assistants vs the older and more cumbersome chat method.

OpenAI - if you are listening, provide us with an update please.

[willer]

I would love to hear an update as well. The Assistants API hasn’t moved forward at all since launch. Did the person who wrote it leave the org or something? Should we all abandon this? ChatGPT is clearly using something we don’t have, as it has streaming, assistants, retrieval, and code interpreter.

[RonaldGRuckus]

Great comments.

There has been no noticeable changes to the Assistants Framework for almost 1/3 of a year since inception.

It makes no sense to me. Does ChatGPT/GPTs not use Assistants? Was that not the purpose? It has all the features I expected in either a decent amount of time, or at least with a general idea of a roadmap.

Assistants is conceptually amazing, especially for people who want to combine a bunch of OpenAI products. A lot of people bash it, for good reasons, but the reasons are temporary.

So OpenAI. Wtf. Your walled garden is just a bunch of dead, neglected plants with a fire in the corner.

Can you guys just let us control the threads in the meantime while you snail your way to progress?

Token management is a joke. Your context management is ridiculous and kills the potential of using 128k GPT-4. Retrieval is insanely limiting and expensive. No streaming, no roadmap, no updates.

I wouldn’t care for any of this if I could just have the option to control threads & add assistant messages.

[logankilpatrick]

We have lots of stuff in the works (including streaming) with more to share soon! Thanks for being patient with us, hopefully it will be worth the wait : )

[danny-avila]

Legend. Thanks for always being in touch with the Dev community!

[duncanhaywood]

@jhakulin

course on your scenario if there are those intermediate messages or not. E.g. when creating charts with

I found the latency on chat completions to be more performant on the assistants api even taking into account the perceived difference in streaming (ie timing the time from request to last token in the streaming response).

[Robs]

Thanks Logan - How soon can we expect to hear?

[logankilpatrick]

Can’t share the specific timeline but we are definitely at the tail end of the process.

[audioone]

Thanks. I know you guys have a lot on your plate like Sora. But, please don’t forget us developers, we are a part of your revenue stream too - especially if you want to reach that $7 trillion campaign.

Really need streaming. Can’t really deliver a production level solution without it. Don’t want to continue use the onld method of context window management just using gptx.

Assistants API is perfect for everything we need to do because of it’s utility and modularity.

[dpasca]

This thread is a reminder of the dangers of getting platform-locked with some high-level API.
You get a lot quickly with little work, but then you’re at the whim of the service provider, waiting and hoping for a feature or a bug fix.