Hey there, we are building an interactive bot that can interact for a conversation of about 10 messages.
In our current design we use the GPT4 APIs and for every step of the conversation we send the prompt + all the messages up until that point. This is pretty terrible since: (1) I end up paying the cost of the prompt for every user, (2) as conversations continue the price per message increases a lot!
- Are there ways to go around problem (1)? Is it possible to amortize the cost of one prompt across multiple conversations?
- Are there ways to avoid problem (2)? Are there ways to avoiding sending the entire conversation up until that point?