I’ve been using Chat GPT to generate summaries of long texts, but I’ve noticed that the summaries are often incomplete.
Upon further investigation, I discovered that Chat GPT is not processing the entire text, but rather stopping early on in the document. To verify this, I asked it to provide the first and last sentences of the text and to count the total number of words.
The results were surprising, as the last sentence it provided was actually from the beginning of the text, and the word count was significantly lower than the actual number of words in the document.
Does anyone have any ideas on how to resolve this issue? Is this a common problem with Chat GPT-3, or a unique occurrence?
Thanks in advance for your insights and assistance.
Im having the exact same problem. Some time it works just rewriting it but right now it just keeps stopping in the middle of a sentence over and over when doing a relatively long text
But it is more likely that you are asking a question of a model that can handle 4096 tokens - but your prompt is quite long. When the AI tries to respond, it can only use the tokens you have left over
So if you ask a question and provide text that the answer should be based on in your prompt that consumes 3000 tokens (lets say), then the AI has to fit its entire response in the remaining 1096
If you take this to the extreme and provide 4000 tokens in your prompt, the AI has only 96 tokens.
Of course this is a lot worse if you are using one of the models with 2048 token limits
Edit: ChatGPT (not gpt3) appears to be based on a model of 4096 tokens (derived from this post)
Correct, at this time 4096 tokens is the limit on the ChatGPT model
The usual recommendation I see is summarizing chunks, which isn’t the best way…
It’s a matter of compute / processing power and serving the model to millions of people on a daily basis. I’m confident that over time, the token limit window will be opened…
1 - yes this is correct (but in your example, you used 4067, so there will be 29 tokens left)
2 - At most, a single token can only represent a single word. But on average, a single token is roughly four characters of english writing. In some cases it is not enough to represent an entire word. (Samatha requires 3 tokens, John requires 1, yes requires 1, and no requires 1)
So a single word summary will probably be no good to you - unless you want a “yes/no” answer.
3 - This is very subjective. But, if it was me, I might input 2500 tokens and expect a summary of 700. But you should probably play around with the values to see what works for you (some text will be harder to summarize than others)
So if I understand you correctly, the tokens limitation applies to both the question and the reply? So if I write/paste text that amounts to 2000 tokens, the AI will only have 2096 tokens left for its reply, if that isn’t enough, that’s when it stops and I’d have to write continue?
I thought the tokens limit only applied to the answer.
“continue” won’t work in all cases, it will start its response from the beginning every time for the subsequent prompts. Any other working prompts to tell it to continue from where it is stopped…?