We are exploring various options for integrating chatbot to our product, and It seems assistants APi is the best thing to way forward, but little concerned over the pricing issues as raised by many in the forum.
Scope of our chat bot is for uploading the knowledgebase and allow visitors to ask question/faq to the bot to get answers, nothing more.
As a newbie here, I have some questions to get clarified by experts here.
- For the above use cause, can I save the chat history in the db, and sent only the user input questions every time. (Many people had pointed out about out of context and hallucination issue). Does this way works, or I need to sent back the whole thread/conversation each time which burns the token credits quickly.
- If above doesn’t works, any suggestion to solve the cost issue ( I saw some user pointing about summarizing, can somebody point out how this can be implemented).
Any advise is great for us to start this.
If you’re using the same thread for each run there’s no way afaik you can delete previous messages to save tokens. You could create a new thread with the same Assistant and feed it a summary of previous interactions. This kind of defeats the purpose of Assistants though, they are designed to keep a context of previous interactions, this is why they can be very useful. If you want to implement your own memory retrieval system with a db etc then maybe not use Assistants api? Cost is a bit of an unknown at present but I know the cost soon adds up.
Our use case is a simple faq bot, to eliminate the man hours spent for repetitive questions asked by users. So i guess even if we use same thread, it wont burn much tokens. We plan to store the history in db for other purpose too. my worry is how economical will be this for local business owners (which is our target users) to setup bot for eliminating the human hours spent for answering same questions.