According to OpenAI The latest GPT-4 Turbo has a knowledge cutoff of December 2023. When I asked GPT-4 Turbo in the API, it claims to have the knowledge cutoff of September 2021. What’s happening, why is the knowledge cutoff reversed to the old GPT-4?
2 Likes
Likely hallucination.
What’s your system prompt?
Nothing, I choose an empty system prompt. Also this happens close to everytime. It tried around 10 times and it always said September 2021.
It doesn’t know it’s knowledge cutoff date, how could it?
1 Like
Until recently, it was correctly answering about the knowledge cutoff, but lately (perhaps with the latest model), it not only seems unaware of the knowledge cutoff but also claims to be based on GPT-3, although it’s true that GPT-4 is based on GPT-3.
This sentence may have characteristics unique to AI, but I assure you it’s not from a bot.
The AI will answer when you provide it the knowledge of its own knowledge.
2 Likes
I understand that the model is unaware of its own knowledge cut-off, but the consistent responses regarding this cut-off might be causing confusion.
Upon rechecking, the cut-off of the knowledge that the model responds to suggested unreliability, yet it did not deny the possibility of knowing things beyond that. However, most of the people conducting these checks might be doing so in English.
If this is truly being implemented for all language speakers, similar responses should be expected regardless of the language in which the question is asked.
GPT-4-Turbo basically has all the quality of a random word generator at this point.
Above, the chance of a token is halved just by adding the word “assistant” within the system message. Probabilities at that token position:
January: 63.339285%
September: 36.114978%
(space): 0.515429%
early: 0.015157%
August: 0.006921%
December: 0.005903%
October: 0.000846%
April: 0.000467%
November: 0.000275%
March: 0.000259%
February: 0.000107%
June: 0.000091%
the: 0.000076%
July: 0.000066%
Jan: 0.000063%
May: 0.000021%
fall: 0.000010%
september: 0.000008%
It has clear tuning on two - and will use them in wild variation.
April and December are way down there…
1 Like
I’m sure you’ve totally got what I’m trying to say.
The knowledge cutoff date for my training is September 2021. This means that any events or developments occurring after that date may not be included in my responses unless updated information has been integrated into my system.
Probabilities at “September” token position is…
September: 78.84%
(space): 13.07%
up: 0.079%
no need to write more probable token…
Should I put spaces in the system message??
gpt-4-1106-preview knows it’s own cutoff pretty well. I asked around 10 times and it was all 2024/4. Older GPT-4 Models also. What is even going on? Sometimes I ask the new GPT-4 Turbo some info on events that happended in November 2023, and it still doesn’t know.
1 Like
I really haven’t found any evidence of concrete knowledge past April last year. Maybe a bit about chat completions API call usage that would be on the cusp.
Providing a system prompt with knowledge cutoff can spurn the AI into answering newer facts correctly or even hallucinating if going into the future, but this message doesn’t unlock any more insights about trains:
“You are ChatGPT, a large language model AI assistant trained by OpenAI, powered by GPT-4.
Your training on knowledge includes information about events and developments up to 2023-12.”
Solving life’s problems rarely is “be a search engine”.
1 Like
ChatGPT [GPT-4] said December 2023? GPT-4 Turbo and ChatGPT+ using same technology?
(I know the deference between 128k Windows contexts)
I think you are showing that the tuning of AI models on just prompt and not a system message, as would be done on instructGPT completion models and initial ChatGPT, is predominantly 2021. My example screenshots alternating between between two months were a contrast between two system prompts.
Fun with AI: simulate early ChatML on -instruct tuning
Include “GPT-3” or “GPT-4” and you can flip the answering to 2020 or 2021 (just by AI pattern inference of GPT-2=2019).
(an AI model which was retuned on a subset set of RLHF and to make a stop token regardless.)
2 Likes
Now ChatGPT [GPT-4] saying, “I’m not GPT-4, I’m Based on GPT-3.5 architecture. My knowledge cut-off is January 2022”
Bro, what is happening with ChatGPT Server’s?
Are you sure? ChatGPT plus GPT-4 is based on GPT-3?
GPT-4 is based on GPT-3.5, which in turn is based on GPT-3.
All models in OpenAI’s GPT series are based on their previous versions.
This means that saying GPT-4 is based on GPT-3 is not necessarily incorrect.
Of course, it can be confusing, though.