Is there a preferred method for state retention in longer conversations? Once a conversation chain gets too long should I summarize the on-going conversation along with the previous summary, are there more effective methods?
This is a nontrivial problem that I’m working on tackling. I tend to agree that summarizing the past conversation is the way to go. In fact, this is how human memory works. The older the memory, the simpler the representation is. Fresh memories are very clear and declarative. Older memories are simple impressions. I draw lots of inspiration from neuroscience. Check out this article: Memory Details Fade Over Time, With Only the Main Gist Preserved - Neuroscience News
4 Likes
Yes, this is a hard problem. Summary is certainly a good start. Wish I had twice the space for prompts though as it’s really really hard to cram everything in.
Even if we had 10x the space there’s still going to be a need to handle larger amounts of data. Some of the problems I’m tackling would benefit from inputs of several million tokens but obviously that’s a ways off. So in the meantime I’ll need to compress the information somehow.
Even humans can do this because we have limited working memory, so clearly there are some tricks we can learn from neurology.
2 Likes
I had thoughts about that too recently… I agree with summarizing as an easy start but even that will start to become a problem eventually and it is true that humans work like that, but I want the AI to remember important things better than most people do. I was thinking about maybe picking out key variables in a conversation and keeping those and everything else just gets a basic summary?
Maybe if you ask the AI to recall something that is not in the current prompt, it can scan an external database of previous prompts and key data that I mentioned earlier. It could find the most relating topic(s) and ask if that is what you wanted. If it is, it can inject that data into the current prompt.
I am not sure if that made a lot of sense or was helpful but those are my thoughts up to this point.
OpenAI recently published a blog about this topic. Summarizing Books with Human Feedback
They call it “recursive decomposition”. Basically, just repeatedly summarizing. Essentially multiple pass verbal compression. This is good but it underscores how humans can keep (and retrieve) far more information in our heads about any given topic and spontaneously recall much of it.
1 Like