After working with the beta of the Assistants API (Node.js SDK) for around a month or so, some random nuggets of feedback.
Overall, loving the API design! Here’s a few improvement ideas that have come to mind:
1) Streaming / webhooks support
This one is obvious and I believe already acknowledged by the OpenAI team, but would be a really really nice QOL / scalability update.
2) Ability to store messages in a queue
Currently if trying to add a new message while a previous run is in progress, the API returns an error and I have to implement some kind of message queue on my end to handle adding the message once the thread is free again. A few ideas for how this could be improved:
- Option 1 (in a perfect world): I would be able to just add new messages to a thread AND the currently active run would be able to take them into account
- Option 2: The messages would be added to the thread, even if not taken into account in the current run. Then, once the current run is finished, I would be able to trigger a new run to handle the new messages.
3) Ability to pause / continue / restart runs
This one is interesting, and is related to making these AI assistant chats feel more human. If we think about how a typical human-to-human chat works, it’s often the case that people send their messages in multiple parts, and the person responding will wait until they feel like they have enough information to respond - based on signals like whether the other person is still typing or not.
For example a pretty normal conversation like this:
John: Hi!
John: Was just wondering if you could help me out with something quickly
Mary: Sure, what’s up?
John: Something’s wrong with my car
John: I can’t seem to shift into reverse
John: And there’s this huge blinking light
Mary: …
Would work a bit weirdly in “AI assistant land”, as the bot will automatically respond to all new messages instead of being able to look at signals like whether the other user is still typing before starting to formulate their response.
For these use cases, it would be very useful to be able to for example:
- Pause a run if the person is still typing
- Once the user sends a new message, add it to that run and then continue it
4) Ability to add assistant and “narrator” messages
Currently we are only able to add messages with role: "user" but there are many many interesting use-cases for being able to add also different types of messages to the chat.
Assistant messages: in some cases I’d like the assistant to send a message to the user, without it being a direct response to what the user just sent it. For example, some kind of status update on whatever was being discussed. Of course if building our own chat UI, we can add these sorts of messages without them being part of the OpenAI thread, but it would be important for these messages to be present in the thread, so that they can be used as context for future assistant runs.
System messages: there are some interesting cases where we’d like to be able to add “system messages” as context to a chat, which inform the assistant of some kind of status update and might potentially trigger it to send a message to the user. For example:
— |Discussion about an order] —
System: The delivery status for order #123_456 has changed to IN_TRANSIT
— [Trigger a run on the thread] —
Assistant: Good news John! Looks like your package is on the way - remember, you can always check here for status updates: https://usps.com/track/123_456
This example case could be solved by either of the above:
- Adding a system message and then triggering a run on the thread
- Generating the assistant message in some other way, and then just adding it to the thread
But I feel it would be very useful to have both options available!
That’s about it - also interested to hear if you’ve ran into similar needs, and if you’ve found good ways to work around them with the current API offering 