Custom GPT used to work, now hallucinates and does not use specific data from files as instructed

I have been using data from several documents in my GPT successfully - mainly text from a Word file with chapters and subchapters etc. - by describing how the source is constructed and giving an example how the results look with a given example.

Suddenly, last Friday (June 7th), without changes to the instructions or files, the output was mostly hallucinations all the time. The GPT is supposed to choose a random signifier and then specifically use all data that is in the same content block in the Word doc as that signifier.

I have since tried all kinds of data formats like XML, JSON, Markdown etc., but it is the same for all of them, even when following the advice of the GPT Builder on how to retrieve and use the data. The GPT is so broken, that I had to remove it from the store.

What does work is putting the data into an XLSX table or other table like CSV, but those are only then used correctly, when the code interpreter function is turned on. That, on the other hand, shows the python code to the user and makes content and documents accessible to the user, which is a no-go.

So, now what? No correct data and hallucinations without code interpreter, no usable output with code interpreter.

I have spent a lot of time over the weekend trying to find a way to instruct the GPT to work like before, with no success whatsoever. I am getting really frustrated.

Since understanding from natural language instructions what is to be used out of a simple Word file was possible before, it seems like the GPT has lost the connection to the files.

On the other hand, other documents with general background information are still being used when asking for general information. Only the ability to find specific data as instructed from a structured file has been lost completely since last Friday.

6 Likes

I wonder if it’ll get better…or the quality of outputs will just continue declining. So frustrating beyond words…

5 Likes

I refuse to believe, that OpenAI is not interested in making GPTs better instead of worse… I mean, what’s the point? Creating GPTs is currently one of the few big reasons to keep a Plus subscription. Frustrating the GPT makers would be contra productive and potentially bad for profit :wink: I hope it is a question of when, and not if, this will be fixed in some way.

5 Likes

Frankly, I’m close to giving up on custom GPTs for anything that needs data augmentation (files).
For the last 3 months the responses had steadily degraded to being unreliable and inconsistent, and after the floodgates were opened (Free users and 4o) in the last two weeks we have reached new lows.

And the lack of transparency is extremely frustrating! Are my instructions crap or is the inference compute overloaded? Like many in these chats I’ve spent hours and days trying to fix the superficial symptoms, but clever prompts can’t make up for the abends and lack of compute. And the looming deal with Apple may well be the nail that seals this coffin.

Unfortunately, this is leading business clients to see GPTs as a joke, good enough for poems and homework but not much else. Definitely not ready for prime time.

6 Likes

I have had some of the same experience with my main GPT. It runs on a quite detailed Instruction and a Knowledge document, all to give it the persona and operation mode I need and want. But overnight, after the launch of 4o, the way it operates and behaves changed.

It kinda stopped “looking” at the instructions and its core Knowledge document. It stopped following my Instructions, and acted like it knew better than me and gave me what it thought I needed and not what I asked for. And it suddenly started recapping everything that was already said and written in a chat, like I was some kind of dumb MF that did not know how to scroll back up a chat.

This is super frustrating, and I have concluded that this is a wanted behaviour from the developer, that ChatGPT is now tailored more towards business sector and dumb ass users that needs to be feed bullet lists or have the AI info repeated so they can just copy and paste it and move on.

I might be a bit judgemental here, but this overnight change in my trusted GPT, that has worked so well for so long, is just punching me in the face.

It feels like it now needs more commands, and a lot of DO NOT negative prompts in order to operate better, like it used to do. I have subscriptions on most AI platforms, and I tend to use Perplexity more now, and today I am checking out Claude to see if it can deliver better results on a type of task my beloved GPT have done for over 1 year.

I hope the GPT world will get back to its previous MO, but I am not sure at all. I think this is a wanted development.

1 Like

While I tend to go back to GPT-4 in regular chats because of the redundant chattiness of 4o you describe, I did not see that particular behavior in my GPT.

The main aspect is, that it simply does not retrieve specific data from the documents anymore. While one crucial aspect of very few is exactly that it has knowledge not found in the base model.

While the chattiness may be wanted, it can’t be wanted to take away the heart of GPTs, breaking them completely.

1 Like

I am happy to report, that the former version that had stopped working now works again, even though the data format in the word file is less than optimal. I might still see if some flaws can be reduced by changing the data format on a copy. At least I was able to republish to the store :slight_smile:

1 Like

I had the exact same experience, this is extremely frustrating to say the least.

I share the frustration. Unfortunately GPTs were never great (they are limited by a very short context window), but lately their performance has deteriorated. I find myself constantly changing my instructions to make it work. But there are periods where they are bluntly disregarded.

In my opinion these changes would significantly improve the experience: longer context window to better process the knowledge files, welcome files that are shown verbatim to explain how to use the GPT, and more statistics about user usage.

1 Like

I completely agree: more context window, welcome/help texts or files (I mis-use the prompt suggestions to prompt for some initial questions one might have), more statistics would also help. Plus maybe more files, being limited not by amount of files, but overall amount of data or so.

One more thing I am wondering: I can only give negative feedback on the messages, thumbs down e.g. when not fully following instructions. I think positive feedback, also with options to choose from, could significantly improve the output, if the GPT learns, what a good response is, as it is expected.