Hi Everyone,
I’m currently working on a project that involves using the Vision Open API. However, I’ve encountered an issue with counter questions where OpenAI doesn’t recognize older messages. I discovered a solution, which involves sending all messages in context repeatedly, but this increases the token size every time I ask a question. Could you please suggest or help me with this issue?
There’s no way around that currently, you must send a “window” of prior messages each time you call chat completion so that you can maintain a sensible discussion.
Thank you for your response, but I believe OpenAI should consider adding a Unique ID feature to manage each chat session differently. This could be highly beneficial in the future.
Their current solution to that is the Assistant API.
However, that can get a bit pricey and I think the general consensus is that it is a little bit too much in its infancy. It also doesn’t yet support the Vision model.
What you are describing (adding all of the conversation history) is how chat models work. They can provide continuously better answers by utilizing all of the previous conversation history to further constrain/refine the parameter space used to generate the next answer. They can only do this if the subsequent request contains all of the previous conversation.
You can think of the chat history itself as a collaborative process between the user and the model to continuously refine the next answer. You aren’t “wasting” the previous tokens by resending them in the chat history, you are exercising them to get incrementally better answers in the next exchange.
They aren’t truly wasted unless you switch conversation topics frequently (and even then they can play some role in determining how the model responds), in which case, your UI should work out some manner of flushing the chat history and restarting the conversation to manage cost.