Hi K, Thanks for sharing your experience on this. Are your assistants STT or text-based? Do you envision a way to reduce latency to 3 seconds? I am trying to assemble a conversational assistant with voice and in the GPTs from OAI and using the assistant with file search is driving my latency to 80 seconds per answer. I am assembling this through n8n. Any word of advice?
Our assistants are text-based. I don’t feel like I have much to offer you in the way of advice. My sense is that we are still very early in a rapidly evolving tech landscape and that things are likely to get much faster and better in the coming months. It could be that you are just ahead of the curve far enough (e.g., trying to do complicated RAG stuff on top of voice) that you’ll have to wait for the world to catch up to you. Meanwhile, you can focus on functionality while you wait for the speed/performance to catch up. Sorry I can’t help you more.
- Using *-mini helps, but cannot always be enough.
- Reducing the token counts, (per output, window, whatever you can) helps too.
- Create thread and run in one shot
They are still slower. Its obvious.
Switched to the Completion API and never look back - it’s much faster. I’d rather spin up my own code interpreter instance, maintain a different RAG setup and develop agentic frameworks like SWARM using completions API.
Can’t rely on OpenAI to make Assistant’s stable(complained about this a year ago).
+switching to the Completion ensures vendor agnosticism, you can easily switch to other providers for different use case. (maybe Gemini for small, quick tasks)
The main problem of switching to Completion API is the lack of memory of the conversation, which can be workarounded attaching the whole conversation to every message, but obviously it does not really scale in terms of tokens usage and payload.
Is there anyone from the OpenAI Team who can provide a rough timeframe for the Assistant API to be ready for production?
same issue, 6-8 sec response times for assistant api. comment for reach.
Is there anyone from the OpenAI Team who can provide a rough timeframe for the Assistant API to be ready for production?
That would be a big “never”. They have already depreciated it. You have a year to migrate to the responses api. But that doesn’t offer much control over the context of a conversation either.
I had the same problem as you and ended up building my own provider-agnostic solution. Took a fair amount of work, but I have total control over conversation context, tool pre-filtering, and storage to database or firestore. I’ve actually been able to start licensing it due to the demand for an out of the box Assistant solution.