Are assistants the future?
Currently they have some serious limitations (cost, opaque handling of context, little control over threads in general). But the basic structure of building context management into the API is an appealing one.
Anyone know which it is: (1) Assistants will always lack control over threads and context or (2) they will evolve to provide a lot of built in control over context?
Obviously they will evolve since right now they are in beta.
I already find them very useful and use them internally every day for a lot of different tasks.
I feel the current biggest features missing are around files (I would love to be able to upload JSON, CSV etc) and filenames. But the concept and implementation come of as pretty solid.
I think we are facing a reality that we will never have enough context tokens or output tokens for āall our dreamsā but having a persistent threaded API that handles function calling pretty well makes a lot of practical tasking applications pretty easy to implement and allow to focus on a) connecting the systems you want get data from or to and b) perfecting the prompts to execute the tasks.
So I am pretty sure that 2) is the answer., the will evolve.
They feel rushed with way too many oversights to be considered production-ready.
As @jlvanhulst has said, they will evolve. I imagine they are collecting a lot of information from their GPTs (conceptually the same as Assistants) to determine how to properly implement these features, and hopefully soon include GPT-4Vision, Dall-E, TTS, whisper, and some sort of control over retrieval & token management.
I think the fact that thereās no documentation regarding any of these features is simply because they donāt know what to implement just yet.
100%. Itās going to get better, and they (hopefully) will ensure that they always play well with future tool changes/upgrades.
So far according to their documentation they āplanā to āexploreā better control:
We plan to explore the ability for you to control the input / output token count beyond the model you select, as well as the ability to automatically generate summaries of the previous messages and pass that as context.
There has also been undocumented endpoints to delete messages so Iām hoping that in the near future we can have better control by deleting messages, and adding new Assistant messages.
Yeah I really hope you guys(@ jlvanhulst, @RonaldGRuckus ) are right. Assistants feel like the natural next step as does an API where threads are build in (and over which there is considerable control for managing, pruning, and of how they are used to form context).
My bet is the days of roll your own management of conversation history are numbered (or at least limited to special cases).
Checkout my post Right Semantics for Assistants API
I think they are a great way to get validation of a product. Iāve used to build something that started generating revenue. Now iām plannign to move on to my own RAG model so I really appreciate the api for what it is.
Assistants api gave me what i needed for my project/app, i like them.
What iād really like to see is more management of how messages are used to build context (control, visibility, etc.). The basic idea of having context held at the backed, accessible through the API makes good sense (for some scenarios) and is one Iād like to see elaborated.
Agreed. I choose to believe that OpenAI is aware of this as well and not consciously preventing us from having any sort of insights.
I have my fingers crossed for a big update soonish (with GPT marketplace maybe?) That does all that weāre asking forā¦ Hopefully a little bit more.
For nowā¦ 
Assistants are cool, in theory. But this whole constant polling to find if there is a response is an issue, mostly because we donāt know if each poll is costing anything, also, super inefficient in computing costs. Pretty much the whole issue in general is cost. Sure there is the basic in/out costs in the docs but no way to know whatās up. Having to grab the entire message history just to get the new messages is also ridiculous. For now, the chat completions endpoint is way more functional and understandable.
They have indicated in the docs that they are planning to introduce better options:
During this beta, there are several known limitations we are looking to address in the coming weeks and months. We will publish a changelog on this page when we add support for additional functionality.
- Support for streaming output (including Messages and Run Steps).
- Support for notifications to share object status updates without the need for polling.
- Support for DALLĀ·E or Browsing as a tool.
- Support for user message creation with images.
Iām not sure what Dall-E or Browsing means. Time will tell.
Nice! Can you share a link to where you found this? Is it in the blog or something?
Yeah, sorry about that
https://platform.openai.com/docs/assistants/how-it-works/limitations
@Orome You make a solid point about assistants current limitations.
*AI assistants definitely seem like they could be part of our future. While they have some issues now with cost and keeping up conversations, technologies tend to improve rapidly over time. Something like ChatGPT has come so far in just a short while.
*As long as companies keep refining how assistants understand context and multiple discussions, I could see us being able to have really natural back-and-forths with them eventually. Thereās a lot of potential there if the technical challenges get addressed.
Quite frankly, a lot of what can be achieved with Assistant can be done so with regular prompting or just custom instructions.
Where Assistant really shine is in the narrative and the developer experience. Few things it succeeded in:
@raymondyeh
I believe you are confusing Assistants with GPTs.
Totally. In all the dimensions. A user could voice call an Assistant, and discuss what they are seeing on a page, and have the assistant perform functions without having to code an insane amount of logic. It seems like this is the current race in the multi-modal LLM industry.
It reminds me of Firebase (Google). One could create their own authentication, live database, server for function calling, low-cost storage, but why. They have done an awesome job putting it all together to get things going. Then, of course, detachment is still a possibility in the future when money starts rolling.
@RonaldGRuckus
I did not - assistant is the GPTs equals for devs.
If you notice, most use case for assistants could be achieved with existing chat completion already. However the āconceptā of assistant led to a mindset shifts amongst developer that they can add additional knowledge and custom instructions to make gpt-wrapper apps - something they could already do with chat completion anyway.
Assistants are not available in any app store so Iām not sure what you mean here.
I donāt think anybody needs to be AI-trained to understand the concept behind truncation or summarizing. Iām hoping Assistants comes out with both of these options so we can just specify a token limit.
For sure. Iām thinking that their current form of retrieval is by no means what they intend to be considered usable. The more I have been a part of the OpenAI ship the more I have learned that they release things, gather a lot of statistics, and then make the decisions.
I donāt think developers have gone through any āmindset shiftsā in the way you describe though. Personally, Assistants has completely expanded my horizon from simple text-based chat to now match the multi-modality that GPT offers.
I think people who can scrap together some Python and over-extend themselves get bopped (naturally) and wonder wtf went wrong. After getting bopped they can have the respect for Assistants and either move back to GPTs (not a bad decision), or keep coming back and learn a new skill & maybe passion.
I can agree with this for now. Itās important to remember that itās in beta and more features are planned for itās full release. If it has all the functionalities that OpenAI offers this would be same argument for not using a library or service such as Firebase.
Itās in the name: Chat Completion. If your agent is simply for text, then of course you would use Chat Completion. Itās important to be forward-looking for Assistants by what they intend to be.
Bruh, if you donāt want to participate in the converstaion, then donāt. There is no need for chatGPT generated responses here. Pushing AI responses here really is uncouth, and totally unnecessary.
Indeed, any service can be built from components at a lower level, and any API can be created from lower level APIs, just as we could all still be writing in assembly. The question is really about what the most effective and powerful abstractions are. It feels to me like Assistants point the way some important ones, particularly managing conversation history, and use its in forming context. Those seem like things that are best handled as part of the same service that performs inference, rather than by the caller.