I’ve been working extensively with ChatGPT and noticed a potential limitation in its ability to recall the previous exchange in a conversation. In my use case, it appears the model’s retention span is between 10 to 16 interactions.
This limitation poses challenges, especially when relying on the model to recall and understand the context of an ongoing conversation. The ability to maintain context over longer interactions is crucial for numerous applications, and I’m trying to find ways to optimize or work around this.
I’m curious if:
Others have encountered this limitation.
There’s a specific reason behind this limit. Is it due to model architecture, memory constraints, or something else?
There is any documentation or insights available that delve into this aspect of ChatGPT.
Any information or guidance would be much appreciated.
The model has an 8196-token context window, which includes your current message, the system message, space to generate the response, any Custom Instructions, and the message history.
Once you exceed this the model cannot retain any additional information.
Additionally, that amount of remaining space that is actually used for conversation history is a choice of the programmer.
The AI model itself doesn’t store a history of what you talked about. There is a database like you see when you recall an older chat in ChatGPT. Individual AI model calls are standalone, and must be told again what you were talking about recently.
There are many ways to enhance the illusion of memory and reduce the token usage of particular turns - or reduce the total quantity passed to the AI model - replaying to the AI just the user input and not the AI response. Summarization. A semantic retrieval of older conversation turns.
ChatGPT’s backend for conversation management, deciding what past conversation is most informative and should be given to the AI model for the most recent question, is a mystery, and we can only see the behavior, not the documentation.