Custom GPT used to work, now hallucinates and does not use specific data from files as instructed

GPT builders
1

I have been using data from several documents in my GPT successfully - mainly text from a Word file with chapters and subchapters etc. - by describing how the source is constructed and giving an example how the results look with a given example.

Suddenly, last Friday (June 7th), without changes to the instructions or files, the output was mostly hallucinations all the time. The GPT is supposed to choose a random signifier and then specifically use all data that is in the same content block in the Word doc as that signifier.

I have since tried all kinds of data formats like XML, JSON, Markdown etc., but it is the same for all of them, even when following the advice of the GPT Builder on how to retrieve and use the data. The GPT is so broken, that I had to remove it from the store.

What does work is putting the data into an XLSX table or other table like CSV, but those are only then used correctly, when the code interpreter function is turned on. That, on the other hand, shows the python code to the user and makes content and documents accessible to the user, which is a no-go.

So, now what? No correct data and hallucinations without code interpreter, no usable output with code interpreter.

I have spent a lot of time over the weekend trying to find a way to instruct the GPT to work like before, with no success whatsoever. I am getting really frustrated.

Since understanding from natural language instructions what is to be used out of a simple Word file was possible before, it seems like the GPT has lost the connection to the files.

On the other hand, other documents with general background information are still being used when asking for general information. Only the ability to find specific data as instructed from a structured file has been lost completely since last Friday.

10 Likes
2

I wonder if it’ll get better…or the quality of outputs will just continue declining. So frustrating beyond words…

9 Likes
3

I refuse to believe, that OpenAI is not interested in making GPTs better instead of worse… I mean, what’s the point? Creating GPTs is currently one of the few big reasons to keep a Plus subscription. Frustrating the GPT makers would be contra productive and potentially bad for profit :wink: I hope it is a question of when, and not if, this will be fixed in some way.

6 Likes
4

Frankly, I’m close to giving up on custom GPTs for anything that needs data augmentation (files).
For the last 3 months the responses had steadily degraded to being unreliable and inconsistent, and after the floodgates were opened (Free users and 4o) in the last two weeks we have reached new lows.

And the lack of transparency is extremely frustrating! Are my instructions crap or is the inference compute overloaded? Like many in these chats I’ve spent hours and days trying to fix the superficial symptoms, but clever prompts can’t make up for the abends and lack of compute. And the looming deal with Apple may well be the nail that seals this coffin.

Unfortunately, this is leading business clients to see GPTs as a joke, good enough for poems and homework but not much else. Definitely not ready for prime time.

9 Likes
5

I have had some of the same experience with my main GPT. It runs on a quite detailed Instruction and a Knowledge document, all to give it the persona and operation mode I need and want. But overnight, after the launch of 4o, the way it operates and behaves changed.

It kinda stopped “looking” at the instructions and its core Knowledge document. It stopped following my Instructions, and acted like it knew better than me and gave me what it thought I needed and not what I asked for. And it suddenly started recapping everything that was already said and written in a chat, like I was some kind of dumb MF that did not know how to scroll back up a chat.

This is super frustrating, and I have concluded that this is a wanted behaviour from the developer, that ChatGPT is now tailored more towards business sector and dumb ass users that needs to be feed bullet lists or have the AI info repeated so they can just copy and paste it and move on.

I might be a bit judgemental here, but this overnight change in my trusted GPT, that has worked so well for so long, is just punching me in the face.

It feels like it now needs more commands, and a lot of DO NOT negative prompts in order to operate better, like it used to do. I have subscriptions on most AI platforms, and I tend to use Perplexity more now, and today I am checking out Claude to see if it can deliver better results on a type of task my beloved GPT have done for over 1 year.

I hope the GPT world will get back to its previous MO, but I am not sure at all. I think this is a wanted development.

3 Likes
6

While I tend to go back to GPT-4 in regular chats because of the redundant chattiness of 4o you describe, I did not see that particular behavior in my GPT.

The main aspect is, that it simply does not retrieve specific data from the documents anymore. While one crucial aspect of very few is exactly that it has knowledge not found in the base model.

While the chattiness may be wanted, it can’t be wanted to take away the heart of GPTs, breaking them completely.

2 Likes
7

I am happy to report, that the former version that had stopped working now works again, even though the data format in the word file is less than optimal. I might still see if some flaws can be reduced by changing the data format on a copy. At least I was able to republish to the store :slight_smile:

1 Like
8

I had the exact same experience, this is extremely frustrating to say the least.

1 Like
9

I share the frustration. Unfortunately GPTs were never great (they are limited by a very short context window), but lately their performance has deteriorated. I find myself constantly changing my instructions to make it work. But there are periods where they are bluntly disregarded.

2 Likes
10

In my opinion these changes would significantly improve the experience: longer context window to better process the knowledge files, welcome files that are shown verbatim to explain how to use the GPT, and more statistics about user usage.

1 Like
11

I completely agree: more context window, welcome/help texts or files (I mis-use the prompt suggestions to prompt for some initial questions one might have), more statistics would also help. Plus maybe more files, being limited not by amount of files, but overall amount of data or so.

1 Like
12

One more thing I am wondering: I can only give negative feedback on the messages, thumbs down e.g. when not fully following instructions. I think positive feedback, also with options to choose from, could significantly improve the output, if the GPT learns, what a good response is, as it is expected.

2 Likes
13

I’m having the same issues. Extremely frustrating… I was having a hundred percent accuracy querying my GPT and now it’s completely off the rails.

1 Like
14

if the symptoms are similar, maybe it will work again soon, like it did for me. My lesson out of this incident is, that when I realize that things, that used to work and should work, not work anymore, even when improving the instructions, that I will wait a little and in the worst case take it offline for a while, until I see, that it works again. I wasted too much time on trying to fix things, that I did not break on my end.

15

I don’t think IMHO it is the 4o that is the problem. What we have found is problems with the versions of Python used for the functions. Particularly in relation to dataclass creation.

1 Like
16

Thanks for the info. That would make sense, because all GPTs were probably switched to use 4o, when they were opened to free users at the latest. And that was a while back. Then, different GPTs seem to have similar problems at different times.

However, I find that there are some good suggestions here on what could be done to improve the functions and value of the Custom GPTs. :slight_smile:

1 Like
17

Look in here.

This seems to also be affecting file reading.

18

I have same problem… please revert to the 4.0 until you get things fixed.
The GPT does not follow instructions and do whatever he wants… the GPT has also difficulties to answer simple questions. It often try to guess what I would want, and don’t answer a direct question. but transform it in something else. Problem is that… when I ask something, it is to get an answer to THIS question… not something else.

Actually I pay… and I have just lost my tool that was working until 4.o has hit me…

Precision: unlike others… I don’t even use one single line of python

1 Like
19

Under the surface the instructions are in part converted to python code, something you can see if you turn the code interpreter on for a test. So, even though you might and can only use natural language instructions, you most surely have to deal with python and thus might experience that kind of error, too, that ra_0929 pointed out.

1 Like
20

Managed to repair everything…
I have taken the hard approach. I have rewritten everything that was causing havoc… so yes, I think the 4o understand things a little differently.