It would be great if someone from the OpenAI team would address this with some transparency. Both 3.5 and 4 have been significantly degraded in the past couple months, I assume for scalability and cost reasons. However, there’s been no acknowledgement from OpenAI as far as I am aware.
Frankly, I would be willing to pay more for access to the better models. 3.5 has gotten far, far faster, but it’s lost the ability to understand basic concepts. 4 has also improved in speed, but has drastically lost comprehension and possibly memory? I’m speculating that perhaps they’ve reduced the model size of both (possibly using an MOE approach as well) and introduced either some kind of dynamic limitation to the context window, or possibly smaller block sizes, to mitigate the quadratic complexity problem with self-attention. Whatever the cause, I don’t bother using 3.5 anymore, and 4 has lost significant abstract reasoning capabilities and working memory.
I’ve noticed that reporting bad answers on 4 tends to significantly improve the quality of the new answer presented. I’m wondering if they’re delegating it to the old (larger?) model when this happens. Not sure on 3.5, it’s just too dumb to be worth working with on complex tasks now - I just assume it’s a consumer/hobbyist model at this point.
If anyone from OpenAI reads this:
The quality loss is dramatic and noticeable - you are absolutely failing to keep things consistent with your changes. You are destroying your competitive and first mover advantages and developing a negative reputation. Just search the web for “gpt dumber” if you don’t believe me.
If you haven’t seen the loss in the data yet, it’s because companies like mine are now actively exploring alternatives for product development and will either migrate entirely or make you one of several models in the coming months. You have competition now, and they aren’t building a reputation for inconsistency.
Even if your quality is comparable or slightly better to other offerings, this lack of transparency, consistency, and options is infuriating your customers.
Be transparent about your changes. Charge me more if that’s what it takes to keep access to the better models. Stop destroying your brand.
Also, let us pay to lift the stupid GPT-4 cap in ChatGPT.