We are creating a chatbot, and we save the conversation history on our side. We need to send the whole conversation history up to the current point to GPT to get response . It works fine but it’s very costly.
With the new Assistants API, we don’t need to send the history, the thread have the history and it will send the history to the model.
I am wondering: does the cost of sending the history to the model still count as cost of the developers or we now just be charged by the message we send to the Assistants API?
Found my answer here Assistants API pricing details per message - #66 by _j.
By reading that, my conclusion is:
Assistants API can actually more costly and it’s hard to control the history we sent to the model.
Assistant API is just a wrapper. The cost is on the model. Assistant API still sends the token to the mode which we need to pay.
1 Like
Yes, I’ve found Assistants API to be decently pricey so far. However, that is a good question, whether we are paying for the context history of messages to be sent back as input, which I assume we must be? Since that is part of the model’s input?
Save the conversation to your db, next time they open it, the bot will start with the knowledge of that conversation. But it will still start from that large accumulated tokenage point in the conversation. This is how I do it and I alert people not to go past like 60 messages on one thread (not the API thread, the thread you save in your db), because the bot can start to hallucinate.
1 Like
Interesting, I haven’t tested the scenario that the conversation passing 60 message. Thanks for the remind that the bot can start to hallucinate. 
I am trying to cut the conversation history to just enough for the model to see the history. Also considering get a summary of the history and use that as history context.