Experiencing Decreased Performance with ChatGPT-4

agreed. To add, the cap is still 25 with noticeable quality degradation. Those two factors make it increasingly unusable and it is now a deeply frustrating experience. At least with gpt3.5, you can infinitely brute-force it to get the correct response.

1 Like

Remember that there is limit on max tokens and for GPT-4 it’s 8192 tokens so if you have 15 messages which are long then old stuff gets truncated.

1 Like

The token limit on ChatGPT-4 is lower (4k). The difference is that in the March model, a conversation could easily hold 25+ messages without any loss or errors. Since the May versions, the context is lost much earlier, and hallucinations occur quickly, after 10-15 messages, sometimes even sooner.
Furthermore, messages are truncated more quickly than before, which requires consuming two messages instead of one within the limit.

Not to mention the issues with reasoning and contextual understanding, the reduced ability to make connections between certain concepts. It’s subtle, but it’s clearly noticeable - I’m not just talking about a few messages, but the entirety of conversations that have become more “dull” or less intelligent.

The speed has been increased (it’s an improvement that is of little use within the context of a 25-message limitation) and it’s likely related to reducing computational costs rather than a genuine intention to enhance the service.

From what I have read here and on Discord, people who use the API don’t seem to be affected by the reduction in service quality.

1 Like

I have been giving it a series of tasks to compare and contrast. Some of these include past conversations and the collection of various prompts I have performed and the responses I had being actually analyzed the positive things I wanted the bot to do versus the performance it was giving

You provided ChatGPT with the following story that was the beginning:

“In a sterile nursery a low hum and whirl is all that was heard from mechanical arms. Each bundle of joy sleeping soundly, save but one. This fussy baby boy know not what his fate will be. Suddenly above his bassinet in orange light flicked on. As attentive as a clucking mother hen, the whirling robot arm whizzed to assess the infant. It began a quick calculation using the diagnostic panel connected to the unsuspecting baby boy. Precisely in 9.18 seconds the orange light flicked to red. Defective. Behind the bassinet with the cutting sound of a blade a panel slid open. The fussy baby begins to wail, maybe having a sense of his impending doom. A conveyor belt pulls the bed into the dark abyss.” :books:

Then you asked it to “get the scene”. :thinking:

In my opinion, you were not using specific or precise language, or enough clarity. :confused:

You did not specify what scene you wanted to get. :man_shrugging:

Was it a scene that continued from where you left off? :clapper:

Was it a scene that showed a different perspective or point of view? :performing_arts:

Was it a scene that explained the background or context of your story? :scroll:

What genre, style, tone, or mood did you want for the scene? :star2:

Who are the characters, what are their roles, and what are their relationships? :raising_hand_man:

Where and when does the scene take place, and what are the relevant details of the setting? :earth_americas:

What is the main conflict, problem, or goal of the scene, and how does it affect the plot? :dagger:

What is the purpose or message of the scene, and how does it affect the theme? :memo:

These are some of the questions that you could have answered to make your prompt more specific and precise. :raised_hands:

By providing more information and context, you could have made your prompt more clear and easy to follow. :pray:

I hope that helps you understand why I think your prompt could be improved. :blush:

According to MasterClass², some steps to write a good story are:

  • Brainstorm for character and plot ideas
  • Outline your story structure
  • Write your first draft
  • Revise your story
  • Edit your story
  • Get feedback
  • Polish your final draft
  • Publish your story
  • Celebrate your achievement

According to wikiHow³, some tips to write a good story are:

  • Make character sheets and choose a story setting
  • Set the scene, introduce the characters, and establish a problem in the first 2-3 paragraphs
  • Fill the middle of the story with action that shows the character(s) working on the problem
  • Present 2-3 new challenges to keep things interesting
  • Create dialogue that reveals something about your characters and keeps readers’ eyes move down the page
  • End with a resolution that ties up loose ends and satisfies readers
  • Use sensory details and vivid descriptions to immerse readers in your story world
  • Show, don’t tell: use actions and dialogue to reveal character traits and emotions
  • Use literary devices such as foreshadowing, symbolism, metaphors, and irony to add depth and meaning to your story

I hope that helps you improve your writing skills. :blush:

Is there anything else I can help you with? :raising_hand_man:

It is getting ahead of the ship allittle bit. Meaning
It can now ask questions about your prompt before you think about these questions. Some one somewhere has programed the algorithms to be focused on precision and clarity, from, humans? It’s been increasingly more difficult requirements to craft a prompt .

1 Like

I completely agree. I use gpt4 to help solve coding issues. Yes it sometimes hits a solution right away, but more often than before, it goes down rabbit hole after rabbit hole, and it definitely does not remember the context as long as it used to. On a few occasions I specifically tell it to avoid doing Thing A, it agrees, then two prompts later it does Thing A. It’s maddening.

A solution that’s oddly worked for me on a couple of occasions is to open a new conversation, restate the problem, and whammo it finds a solution right away. Bizarre.

3 Likes

I’ve also noticed a huge decline in the usefulness of chatGPT, for that reason I won’t bother moving to the paid version.

Not that bizarre if you think about it. By opening a new chat you’re purging incorrect ideas from the context, this can make it easier for ChatGPT to hit upon the right answer.

2 Likes

Yes. But what I find strange is it that it seems to “forget” direct instructions to avoid a certain thought path from three or even two prior prompts. It gets obsessed with certain solution paths. I’ve caught myself wanting to argue with it—come on, we’ve done that and it didn’t work!

And as mentioned, when the cache is truly cleared by opening a new chat, it doesn’t have any problem solving the issue quickly. Like a second option from a new instance of itself is gold.

1 Like

LLMs trend to produce poor results with negative instructions. They do much, much better when you tell them what to do.

So, this should not be strange at all.

1 Like

There is a clear decrease in performance and the speed of GPT4 resembles more and more the one of GPT3.5.

Memory within an ongoing conversation has worsened and after 2-3 prompts it forgets previous messages: unacceptable!
GPT4 was easily able to recognize the context of any long ongoing conversation, even recalling details about many messages exchanged earlier.

OpenAI if this is degrading is due to the cost of GPU or what, we are many able to pay more for having the top model, but for sure nobody here is willing to keep paying for something that has been degraded without any announcement.

What kind of business attitude is this, at least announce it, announce to the world “hey we needed to dumb it down for privacy issues and other useless trivial moral things”

3 Likes

Okay sounds like you know a lot more than I do. When I keep telling it what to do and it keeps going back to faulty patterns, I guess it’s best to just start a new conversation.

Well, that’s a very encouraging thread here from Sam’s visit to Israel:

Yam peleg on Twitter: “# Did the chat model change? The latest GPT-4 made the model much faster. BUT there were claims about the model’s performance affected by this update. I specifically asked Sam about this: Sam denied that any change in the model had been made.” / Twitter

Hi @logankilpatrick
If you need further info to hone in the cause of this issue, I’d say two things based on the summary of this thread.

1- It lays out great plans, but it fails to execute against those plans.
2- When you point it out to check its output against a set of validations, IT DOESN’T do any validation and instead it hallucinates as if the answer is correct.

The last example shared by @radiator57 is a very good one is it clearly demonstrate the issue.

I understand with ‘1’ there are probabilities here, so I’d expect that answers will vary and might get it wrong sometime.
But it is really critical that it should be able to correct itself when asked to, right now it will only correct itself only if you specifically point exactly to where the problem is, by that time, it would acknowledge its error and fix it. But other than that it would keep hallucinating that the answer is correct. I have used OODA loop kind of approach to test that theory and I have 100% repro of that behavior. 100% repro !!!

I think that’s the major concern right now, it lost self correctness and it must be nudged by either a human or when it receives a a response as part of a tool use. That behavior was not there before and it was able to self correct itself just by pointing it out to review its output and it meets the requirements. right now, this seems completely broken.

And thank you for releasing the the new GPT guide, it is helpful. but we really do appreciate you folks looking into that behavior. again I have 100% consistent repro of that behavior if anything needed.

Thanks.

1 Like

No comment really :frowning:
Doesn’t follow instructions, picked up its own sentences to sample, doesn’t understand its output and just hallucinating its way out. this is a very basic example, which basically makes the use of this model for coding tasks extremely challenging since it errors out a lot and refuse/ignore instructions.
Please bring the old GPT-4 back :frowning:
Not sure what further evidence you folks need, in many enterprises, Devs won’t go home unless they review each and every PR that went to the release branch and find the root cause of that regression, unless this was something intentional that was made to it.

1 Like

Doesn’t follow instructions

It is not an instruct model.

picked up its own sentences to sample

I’m not really sure if the context for this complaint as I don’t know what the original prompt was, but often when you ask it to do something it cannot do it will just provide you examples of what doing that thing would look like. This is not new behavior, it’s been that way since jump.

doesn’t understand its output

It does not have the capacity to reference its output while it’s producing that same output. This isn’t something which can be accomplished (at least not reliably) in a single prompt. If you want it to reflect, have it do so after it has completed its response.

and just hallucinating its way out.

Yep, that’s what tends to happen when you try to force the square peg into a round hole.

If you push the model too far outside its wheelhouse it is bound to go wonky. This is not new and should be unsurprising to anyone experienced working with LLMs.

I’m sorry, I’m just not seeing the basis for a real complaint here.

1 Like

It’s totally horrible now, GPT-4 will start looping output of code or other information over and over again.

I ask it to code, it hits a point and does a ``` suddenly and starts again! It’s braindead vs. before.

If you aren’t actually pushing it with what it could do previously, you wouldn’t notice. Yet if you are really using it fully, you see it is obviously much dumber than previous to plugin additions.

2 Likes

I knew this day would come. The days of ads and profit. I had over 2-3k conversations with chatgpt 3.5-4.0 since January. The model clearly went down the drain. Increasing price isn’t the solution. Do some of you understand that if we go down this route it’s a very nasty one?! We are talking dlc level but into your daily life. Wanna be a programmer? Nope, you are fighting against guys who have access to ChatGPT 8.0 Agi-Level @10,000$usd/month.

Back on the subject. When they announced availability of the ios app everywhere I was shock. Customer base just went through the roof. On a technical, it’s currently impossible for them to process all query @ full performance. They simply don’t have enough processing power/ or money. It’s obvious they have reduce the performance of the model to increase speed and number of concurrent user.

6 Likes

just leaving my reply here too so this thread gets more attention, i have been plus user for over 3 months but now i have canceled my subscription as it is no longer worth the hassle, i am mainly working with extensive coding in multiple languages and i have noticed performance decrease weeks ago, but right now OpenAi completely finished it off K.O., it seems they are focused on quantity not quality anymore, sad moment, that means it’s time for alternative options…

3 Likes

The tool is not perfect and everybody here knows and understands

The tool is absolutely fantastic and everybody here thinks the same just because the fact they are connected to this forum

But lot of people have experienced tool a downgrade in quality since May updates. I’m part of these people, and I write it with complete intellectual honesty and serenity.

2 Likes

I’ve also been experiencing decreased performance with ChatGPT 4. Its not just one thing, its a multitude of small differences which dramatically reducing quality. Repetition, especially on correction and disclaimers. Getting simple functions wrong. Not recognizing the incorrectness even when asked to review the code it wrote. Resistance in offering alternatives or exploring ideas (dogmatic). I rarely ever needed a long conversation with GPT-4 because I was so happy with the answers, now I sometimes don’t even follow up because I am so annoyed with its responses.

2 Likes