API response using the context, the previous responses

[jose.rosauro]

I’m trying to keep the context using the API, but I’m having trouble with the conversation_id.

As a workaround, I’m currently saving all previous conversations in Message (Role and Content), and then using that information the response is OK regarding the context.
However, if I have too many previous conversations, ChatGPT’s request limit becomes a problem, so I need to keep my new questions short.

Do you know of a more elegant solution to maintain context in this situation?

Many thanks!!

[jacknoutch]

Here are a few strategies. The first seems to me the most useful, but your application may need be better served in another way.

1. When the conversation grows to a certain length of tokens, get the AI to summarise the conversation (or perhaps the earliest part of the conversation). Replace the messages which have been summarised with the summary and use it as the context input for the next message. The advantage here is that you can keep summarising over and over, so the conversation will keep going. But the longer you go on, the more details will be forgotten. Another advantage is that you can set the max length of a conversation’s set of messages.
2. Alternatively, you could use a model that allows for more tokens. (See here for documentation on the max tokens for each model.) This gives your bot more “memory”, but there will still be a hard limit.
3. Do you need all the previous messages? Could you keep only the most important ones? Testing the responses given by the bot to get only the salient ones could allow you to keep only the most important information in a conversation.

[sps]

@jose.rosauro

conversation_id

There is nothing called conversation_id

Build your own AI assistant in 10 lines of code - Python

Thank you @dmirandaalves Yes it is very much possible to do that by using tiktoken to count tokens everytime before making the API call, if the token count exceed a specific threshold, get a summary and pass it as the system or user message, and then make the call with user input. The approach above is easy to implement, but a more robust yet complex approach would be to use embeddings.

[jose.rosauro]

You are right, I had read it from somewhere, but I doesn’t work for me, and when I searched the official documentation page, there was nothing about it. Thanks!