The ChatGPT API Documentation says send back the previous conversation to make it context aware, this works fine for short form conversations but when my conversations are longer I get the maximum token is 4096 error. if this is the case how can I still make it context aware despite of the messages …

They / we use various methods to truncate, summarize and otherwise insure the tokens count is below the limit. FYI, chat completions from the API contain the token usage numbers and you can track this in your app as your chat session progresses. I update and store the token usage numbers in a DB…

Hi @tzekid I am running a bit behind of finalizing a new chatbot based on the new chat API method due to family priorities and “honey do” tasks around the home, so am not 100% finished yet with a new chatbot based on the just released chat API method. What I have done so far is to create two DB ta…

Thanks for the great response! :slight_smile: I only started playing with the API last night. The only solution I’ve currently spun up is a very simple “summarize convo” function that simply calls the GPT-3.5-Turbo API. The resulting convo then only has one system and one assistant response (with t…

[image] tzekid: GPT should infer the user’s message or context in its answers, right? I’m not sure. It’s something to test, of course. Keep in mind that generative AI does not really “infer”, so not sure what you mean about that. Generative AI is just a fancy, autocompletion engine which ge…

Thanks you guys! I think checking available token and truncating the message starting from the oldest one is the best option (somehow I tried it but I was just sending only the last message with the current one - It didn’t work out tho[not always context aware and would still pass the limit]). in …

[image] tzekid: By the way, wouldn’t it make more sense to remove the prior “role: user” messages? D After working on this for hours today, easily seeing fatal errors for exceeding 4096 way too much. I am going to experiment with stripping out all assistant replies from the API. This 4096 to…

I’ve had some success with summarizing in my tests yesterday. Three things that I noticed: You need to tweak your “summarizing prompt” to get the compression / accuracy ratio you want. (In my testing) For longer conversations I used ca. 10ish % less tokens in total compared to sending the whole c…

fyi, in addition to storing conv history in a local db, langchain does one additional optimization thats worth mentioning. It summarizes past history: https://langchain.readthedocs.io/en/latest/modules/memory/types/summary_buffer.html So, the prompt I am supplying is actually a summary of past his…

ChatGPT api maximum token

ChatGPT

tzekid March 4, 2023, 10:04am 3

Care to share your tricks / methods about how you summarize the messages/convos?

What is the ChatGPT max tokens of ChatGPT public site?

Topic		Replies	Views
I wish that when using the GPT API, it would be possible to have a contextual conversation like chatGPT API	13	7462	April 5, 2023
Need more than a 4097 token call from chat gpt api API	7	3469	November 28, 2023
Getting ChatGPT to Remember Previous Chat Messages Prompting	36	71164	December 8, 2023
Help Needed: Tackling Context Length Limits in OpenAI Models Community gpt-4 , chatgpt , token , rate-limit , openai	8	21370	February 8, 2024
ChatGPT seems to exceed the token limit and not lose memory API	2	3919	December 29, 2022

ChatGPT api maximum token

Related topics