How can I maintain conversation continuity with chat gpt api?

I can’t continue the conversation using the API…
Is there any other way than to save past conversation history in a DB and use it?

1 Like

Hi

Is there any other way than to save past conversation history in a DB and use it?

LLM models today are stateless, meaning that every single bit of context you want it to consider, should be passed with every interaction with it through the messages.

So no, there is no way to maintain a conversation rather than passing it (or its summary) every time to the model.

1 Like

Maintaining conversation continuity with the ChatGPT API involves managing the context of the conversation effectively. Here are some strategies to consider:

  1. Session Management: Utilize the session feature provided by the API. Each session can maintain a context for a certain period, allowing the model to remember previous interactions within that session.
  2. Context Window: ChatGPT has a limited context window, meaning it can only see a certain amount of the most recent text. Ensure that you include relevant parts of the previous conversation in each new API request to maintain context.
  3. Database Storage: As you mentioned, storing past conversations in a database is a common approach. You can retrieve and send the relevant part of the conversation history with each new API call to provide context to the model.
  4. Context Condensation: In cases where the conversation is lengthy, you can condense the context. This involves summarizing past conversations or extracting key points to feed into the API, keeping the context relevant and within the size limits.
  5. User and Conversation IDs: Assign unique IDs to users and conversations. This helps in retrieving and managing specific conversation histories for individual users.
  6. Context Markers: Use markers or tags in the conversation context to denote important shifts in topics or to highlight key information. This can help the model to better understand the flow of the conversation.
  7. Incremental Context Update: Instead of sending the whole conversation every time, you can incrementally update the context with each user input and response from the model.
  8. Fine-Tuning: If you have a specific use case or domain, consider fine-tuning the model on relevant data to improve its understanding and memory in that context.
  9. State Management: For complex interactions, consider implementing a state management system that can keep track of where the user is in a conversation flow or a process.

Remember to comply with data privacy and user consent regulations when storing and managing user conversations.

7 Likes

This might help:

SolrAI: How the chat completion process with standalone question maintains conversational context (youtube.com)

Passing back the conversation thus far like others have mentioned is what’s common, but there’s also the beta Assistants feature that offers this in the form of threads.

It requires a few extra steps to implement. Strictly speaking it’s not much different than passing back the entire chat log yourself, the biggest difference being that the log is on their server.

https://platform.openai.com/docs/api-reference/assistants

1 Like

LangChain helps to abstract some of this away too! :slight_smile:

1 Like

having the same context history added to each prompt from the scratch has some other problems like : duplicate token pricing, and character limit.

Even if we manage to add to history to each prompt (that is the easy part) chat gpt api will charge it duplicately for each prompt. so that may not be the issue.

expanding my example :
I want to import my long text document to chat gpt through api, and ask questions about that document . so if the api does not remember the document, how can I ask the questions about it?

Probably just stored the theadId and associate it with the userId then get threadId everytime user logs in ¯_(ツ)_/¯

If I understand well. We can control the history chat on our dev application ou can use the assistants route to help to organize the bots in a better contexts.

Someone knows about if the effectiveness of assistant is better than use the gpt directly ?

I am building a App that have more than one assistants . And I try to figure out how is the better way to code that.

Tks a lot.