Chatbot for company website to answer product-related questions

API
1

I have been tasked to build a chatbot for a company, it is supposed to be a knowledge base for customers to ask questions instead of writing a support ticket. I already have a JSON file with all the information the Chatbot needs. I am using Assistant with its API. Furthermore, I uploaded the document with the vector_search tool. When I ask example questions in the OpenAI Playground, the response times are fairly decent. But once I try to use the exact Assistant through the Assistant API V2 (Python). The answering times double.

I have multiple questions now:
-Is the Assistant the best tool for my use-case? Is chat completion a better option?
-Could the slow response time be tied to me using the Python library? Could it be faster by using curl and Rust for example?
-Could it be that due to a longer instruction, the response time is slow?
-Is it advised to make your own vector-database instead of using the vector_search?
-Are there any other steps I can take to ensure faster response time while still getting accurate information?

Thank you very much for taking the time to read this!

2

Assistants are more for people who want to get something going quickly. But it’s a bit of a black box, and I don’t think they are optimized for latency and you don’t really have a way to precisely control the context.

I don’t think Python is your slowdown, it sounds like there may be too much context being sent (irrelevant context possibly).

So in this situation, I would use the chat API and create your own vector store. It’s the only way you can really see what is going on, in detail, and you can directly make trades to optimize for lower latency.

Also streaming out the completion is going to speed things up from a user perspective, because they can see results quickly without waiting for the whole thing to render.

So API / vector store / streaming is probably the direction to go to get the lowest latency, and the most control so that you can troubleshoot any latency issues that arise.

I haven’t used assistants much myself for these reasons.

1 Like