That also happens to me pretty much frequently, I tell it not to do something, and it ack’d it back, then I ask it to go ahead as per plan and it does exactly opposite of that.
At one point before sharing becomes available, I gave it a simple Python code to improve, it come out with 5 simple but creative improvements tbh. then I ask it to go ahead and write it, it went ahead and re-wrote the whole thing but instead of adding the code, it just added new functions with single line comments saying that we need to implement this later
What kind of tasks you use for?
If you use it for some simple stuff, this thing is a Ferrari
If you use for some serios stuff, this thing is a Lada 80s model.
would be great to share some chat if possible.
Please note that when I say serious stuff, I mostly means coding, and Python , one of the most easiest language in the world, and also for either small functions or simple scripts, nothing out of the ordinary.
Thanks, Totally get that.
And that was the case before the Plugin model becomes Beta, it used to work fantastically well until the conversation grow and you would re-create a new one as you exactly describe. But it was really doing incredible job and I don’t mind that, in fact I connected it to a VectorDB and can save and restore the gist of any previous interaction with it. So we’re good on that.
What we’re talking about here is a degradation from the get go, for simple tasks and at the very beginning of the convo. Please take a peek at the Chat snippets I shared earlier on this thread.
I have also done the “apple test” and other similar ones, the result is indeed distressing. The most incorrect part of this story is OpenAI’s silence. API users don’t seem to be concerned about the problem, with the payment model being per token, it’s understandable. ChatGPT users are the ones being harmed, the global payment model encourages reducing computational power for economies of scale. API users are also probably a more “professional” population that shouldn’t be upset, while ChatGPT users are perhaps more individual consumers, being treated like cash cows. The fact that there has been a modification to the model is impossible to deny (those who do are either dishonest or blind): the response speed is significantly faster. But at what compromise?
I also noticed the same thing for a couple of weeks now. At times, it feels just like I am using GPT-3.5. And it’s also a lot of other little things that keep getting downgraded. Like the character limit or word count for GPT-4 per prompt is now less than GPT-3.5, which is mind boggling considering we are paying for GPT-4. I’ve decided to just cancel my subscription because it’s no longer worth it and the plugins have been lackluster. Maybe I will buy it back in the future when there are features that actually justify the cost.
These are stochastic models so even with a temp of 0 there’s a certain amount of randomness at play. You could call the model 10 times with the same prompt and get similar answers 9 of the 10 times but 1 of them will be just wrong.
They’re retraining these models on a daily basis to incorporate feedback from a mix of humans and GPT itself. The goal of this feedback is to improve the model but it means that for some answers or tasks they might regress which is what you’re seeing.
I personally think they need to introduce check point versions of the models that aren’t being retrained so that people can build actual applications on top of this stuff (it’s too much of a moving target right now) but that’s just my opinion.
There’s a big difference. It’s just not as good. Definitely not as thorough. There’s a chrome extension “Superpower ChatGPT” which allows you to use multiple models of chat-gpt. I compared and older one called chatgpt-4 with 32k token limit with the current chatgpt-4. - Link to text file
The chrome extension “Superpower ChatGPT” is not affiliated with OpenAI, it cannot give you access to “chatGPT with 32k token limit” as such a thing doesn’t exist
There’s a model called GPT-4-32k which is currently being tested by a handful of people and is only available through the API.
Retraining and updating the model is a costly affair, it’s not done on a daily basis, you can see which version of GPT you’re on in the footer of chatGPT website.
Remember that this is the developer community forum, if you want to talk to OpenAI directly you should contact them through:
– I don’t think he was referring to full training of the model, I think I know what he is talking about which seems plausible to me because fine tuning models to do what you want after you have the base model is not kind of a big deal, I don’t think anyone suggesting that they are retraining the base model here.
– OpenAI online Help is basically useless, at least for me.
Generally they have variety of way to control the response, which IMO happened with the release of the plugins to everyone, which is a good thing but as folks suggested here, this came at the cost of quality to everyone else.
It’s pretty darn obvious
One note though, I see people refer to Bing is being degraded, not for me though and I put it to the test almost everyday.
I gave GPT-4 in the morning 75 line of Python code and asked it to to split to two separate modules, add docstrings and add one more additional parameter.
It took it literally 10 prompts and with each response it missed something.
I just did the same with Bing and it produced a ‘master piece’ from the first attempt, and it actually understood the new parameter I’m adding will require additional changes to one of the function and suggested that change and did it without me asking, took exactly 3 prompts, and 4 more prompts for additional stuff as I got encouraged by its response.
I see Bing keep improving while GPT-4 performance keep declining with each update they release.
GPT-4 (really feel sorry to say that after I experienced its prowess) just spit out garbage to my prompts, whatever prompt technique it just doesn’t work, crazy!
And when you use these stuff, you are looking for it to be helpful and save you time, at the moment it is doing exactly the opposite, I have to use it now when kids are not around so that I feel free to curse as I want.
It can really deceive people with its elegant responses to simple questions, but when you put it to the test, it really struggle even with simple tasks nowadays.
I’m really betting big on Bing when the plugin is getting released because I think MSFT folks will be able to pull it of better than OpenAI. kind of signed up to the developer Beta 10 minutes after their Build announcement and the waitlist site spun up
Good note!,
I’ve noticed that some sites have started blocking traffic coming from chatGPT, that could also be the culprit behind some of the experiences folks are having
