Managing Previous Conversations with ChatGPT

API
, ,
1

Hello,

I am currently developing a website and aiming to implement a chat section using ChatGPT.

The issue I have encountered is that I’m unsure about how to effectively recall previous conversations of a user for ChatGPT.

I used to follow a method where I stored user questions and ChatGPT responses in a database, and I would retrieve them using an approach similar to the code snippet below:

        const response = await fetch(API_URL, {
            method: "POST",
            headers: {
                "Content-Type": "application/json",
                Authorization: `Bearer ${API_KEY}`,
            },
            body: JSON.stringify({
                model: "gpt-3.5-turbo",
                messages: [
                    { role: "system", "content": "{ previous_conversation: {role: 'user', 'content': 'previous chat of user'}, {role: 'assistant', 'content': 'previous chat of chatgpt'}}"},
                    { role: "user", content: promptInput.value }
                ],
                max_tokens: 2000,
                stream: true,
            }),
        });

However, after a while, when the conversations become lengthy, I encounter a “max tokens” error.

Is there a more efficient way to recall previous conversations?
Could you assist me in addressing this issue?

2

Hi and welcome to the developer forum!

The problem you describe is the main issue facing chat systems, in the case of ChatGPT, older context past the token limit is simply not used for the current query, it’s actually surprising how well this works, but can lead to people thinking the system has infinite memory, when it does not.

You could build summarisations of past chat context and try using that or simply truncate older information, see which works best for your use case.

1 Like
3

Thank you for your response!

I appreciate your suggestion about building summaries of past chat context. Could you please provide more guidance on how and where to create these summaries within the conversation flow?

Additionally, how would I go about reminding the system of these summaries during a conversation?

4

You would calculate the size (in tokens) of the totality of the context.

Once that size hits some limit (say, 80% of the token limit of the model,) you would call out to the model, without the user prompting, and prompt something like:
"summarize this conversation, to be used as a prompt for continuing the conversation later" (and each of the previous messages attached)

Now, you store this summary in the database, and wherever you would previously use the preceding conversation, you instead use the summary in the prompt. Once the size of summary plus the continued context reaches the limit, you re-summarize the summary plus the successive conversation.

So, assuming you have a “conversation_id” you’d need two tables:

statements: conversation_id, statement_index, statement_author, statement_text
summaries: conversation_id, to_statement_index, summary_text

You would then generate the prompt history something like:

sum_id = select max(to_statement_index) from summaries where conversation_id = {conversation_id}
if sum_id:
  history = select summary_text from summaries where to_statement_index = {sum_id} and conversation_id = {conversation_id}
else:
  sum_id = 0
history += select statement_text from statements where statement_index > {sum_id} and conversation_id = {conversation_id}

-- now create chat completion based on the history

Obviously this can in addition be tweaked, keeping a few of the last “real” statements even when summarizing, trying to extract main themes to focus on in the summary, and so on.

1 Like
5

Dear John,

I wanted to extend my sincere appreciation for your invaluable advice. Following your method for managing previous conversations with ChatGPT has yielded impressive results. The approach of summarizing conversations to fit token limits worked seamlessly and significantly improved the efficiency of the chat section on my website. :star_struck:

Thank you once again for your expertise and assistance.

1 Like