Does anyone know what is the true token limit on GPT-3.5? It seems to be a lot more than 4096. I read somewhere that it is maybe being achieved by splitting into chunks, and/or summarising the previous chat to get the context for future responses. But, as the below example suggests, ChatGPT is memorizing more than 6000 tokens verbatim.
-
To start with, I asked chatGPT to memorize a number, with an instruction to give me back the number when asked at any point later in the conversation.
-
Then gave it a long text (6112 tokens) to summarise. The summary itself was 608 tokens
-
Then I asked it for the memorized number. It gave the correct answer. (This could be possible if the previous conversation was summarized)
-
Then I asked it to give me back verbatim an extract from the start of the text. I chose a portion which contained mostly repetitive content and not much information content, which is more likely to be removed from the summary/context).
ChatGPT was able to give me back those sections verbatim.
(It is possible that this section was retained in the context)
-
I tried it for a couple of other sections from different parts of the text, and I got correct verbatim responses each time.
(Its unlikely all sections were retained verbatim in the summary context) -
After all this, asked for the original number again- and it had remembered.
Any explanation of how this is working / what is the true token limit?