I’ve been reading up on the API and pricing and hope that i have it right. Please correct me if not.
Open AI doesn’t remember or store any previous sentences so each new sentence being part of the current chat, needs to be sent in the one request so it knows the flow of chat.
Let’s say my app has 10 people discussing the same topic. Does this mean that on the 10th sentence of the chat, it would be sending 55 sentences of however many tokens for 1 person and that would be, times 10 being 550 complete sentences after only 10 different sentences were requested with all previous requests to let Open AI know where it’s at?
If so which I expect it would be, Open AI would not be financially viable after a 30 minute or a 1 hour discussion.
I get that if the topic has changed and the app can know that, it will restart the sentence count but?
You are correct that the GPT models require the entire context to be passed to them each time.
The API has a maximum context length that varies depending on which model you choose. You can manage that yourself, so if you restrict the context to only the last 2048 tokens from any given conversation then the maximum cost will be a fixed quantity per message and you can base pricing on that.
I am not not quite sure what you mean by 10 people discussing the same topic? Do you mean a 10 user multi chat?
Thanks for the confirmation.
Yes that’s for a multi chat. i would be storing all submitted questions in a database as a record so that wouldn’t be hard to recall a few sentences and restart from there. That is if that wouldn’t entirely reset the topic of chat from Open AI? We would need to do some testing to see if the continuity would continue or not.
Sounding like it will be very expensive. Appreciate the info. Cheers.
1 Like