Lack of Consistency, Loss of Quality

ChatGPT
, ,
1

It would be great if someone from the OpenAI team would address this with some transparency. Both 3.5 and 4 have been significantly degraded in the past couple months, I assume for scalability and cost reasons. However, there’s been no acknowledgement from OpenAI as far as I am aware.

Frankly, I would be willing to pay more for access to the better models. 3.5 has gotten far, far faster, but it’s lost the ability to understand basic concepts. 4 has also improved in speed, but has drastically lost comprehension and possibly memory? I’m speculating that perhaps they’ve reduced the model size of both (possibly using an MOE approach as well) and introduced either some kind of dynamic limitation to the context window, or possibly smaller block sizes, to mitigate the quadratic complexity problem with self-attention. Whatever the cause, I don’t bother using 3.5 anymore, and 4 has lost significant abstract reasoning capabilities and working memory.

I’ve noticed that reporting bad answers on 4 tends to significantly improve the quality of the new answer presented. I’m wondering if they’re delegating it to the old (larger?) model when this happens. Not sure on 3.5, it’s just too dumb to be worth working with on complex tasks now - I just assume it’s a consumer/hobbyist model at this point.

If anyone from OpenAI reads this:

The quality loss is dramatic and noticeable - you are absolutely failing to keep things consistent with your changes. You are destroying your competitive and first mover advantages and developing a negative reputation. Just search the web for “gpt dumber” if you don’t believe me.

If you haven’t seen the loss in the data yet, it’s because companies like mine are now actively exploring alternatives for product development and will either migrate entirely or make you one of several models in the coming months. You have competition now, and they aren’t building a reputation for inconsistency.

Even if your quality is comparable or slightly better to other offerings, this lack of transparency, consistency, and options is infuriating your customers.

Be transparent about your changes. Charge me more if that’s what it takes to keep access to the better models. Stop destroying your brand.

Also, let us pay to lift the stupid GPT-4 cap in ChatGPT.

2

First, you are assuming facts for which there is no evidence.

I, personally, have noticed no decrease in performance and I am using ChatGPT on a daily basis. The vast majority of the complaints I have seen regarding the “loss of quality” you are referring to relate to OpenAI’s ongoing alignment work. People seem to be very upset they can’t make the model spew hateful and racist text or deny basic science.

There has never once been anyone I have seen who has provided any actual evidence to support the claim the model quality has been degraded. The vast majority of such claims can be explained by the natural variability in the quality of responses, it is a stochastic model afterall.

You seem to be missing the one, obvious solution to your problem…

Use the API where you can set your own system message and have complete control over all of the model parameters and you are free to pay for as much as you want to use.

3

Well, we cant show the difference between the earlier models and the current ones because we cannot access the older version of chatgpt.

I use it for coding, and it is very noticable how bad it became recently with keeping track with the contect od the problems I give to it.

(Talking about gpt 3.5 turbo btw, gpt 4 is pretty good, but the 25 messages limitation every 3 hours its a bit annoying at time)

4

Closing this topic.

Please consider using this topic instead.

GPT-4 has been severely downgraded (topic curation)

1 Like