GPT4 and Spelling Mistakes in Coding Tasks and JSON outputs

API Feedback
1

Is it just me or is the new version of GPT4-Turbo and GTP4-o really terrible with spelling mistakes, especially in coding. I have used GPT for coding almost since there was GPT3 and it has been consistent most of the time.
However since the release of GPT4-o, the price reductions - it really feels like the quality of the outputs has declined. Tasks GPT4 used to do with ease, its struggling with and often making really bad spelling mistakes. A lot of my use cases are struggling now because even with json outputs – the keys are spelled differently, making the json object impossible to process downstream.

I will share a real example of code from yesterday, with GPT4

    if created:
        # Get the current users of the ChatBot after it's created and the creator has been added
        current_users = set(instance.users.all())
        # Get all users from the profile, excluding the creator
        profile_users = set(instance.profile.users.exclude(id__in=[user.id for user in current_users]))
        # Add the missing users to the ChatBot
        if profile_users:
            instance.users.add(*global_users)

In the code above, it named the variable “profile_users” and even checks for it correctly, but a line below it adds “*global_users” – a new variable it just made up. When querying what is wrong with this code, it can immediately pick up its mistake and tell me – sorry, it was a typo ---- but I find, its making more and more of these type of basic mistakes, meaning I must check everything, making it hard to use it for automation and within coding workflows.

One of our internal apps has been rendered useless because of the rates of spelling errors in the json outputs, we cant use it anymore. We have to go back to manually entering everything and checking for spelling.

I really wish OpenAI would keep older models running even after releasing new ones and make it optional if we want to continue using the older model. Usually after changing a model, we have to rework all the prompts and sometimes we never get it to where it was before.

3 Likes
2

Hi!

Regarding the spelling errors, yes, we have seen a few reports about this in the community but mostly with regards to 4o. This is something we are all more or less managing at the moment. Unfortunately, I haven’t seen any real solution from a user perspective.

Some general advice we have gathered is to be more explicit and use shorter prompts, but this barely addresses this specific issue. I expect we need to wait for the deployment of the next version of the model.

In the meantime, since you’re an API user, why can’t you go back to using an older version of GPT-4 without ‘o’?
This seems like the way forward should be clear, but this is based on my impression that using the older models works better.

3 Likes
3

I’ve noticed the same issue using the web UI, and found your post after googling GPT4 typos.

My workflow used to be:

  1. Copy my existing code snippet into GPT4 and tell it what i want changed or what isn’t working
  2. Copy / Paste back into my project with 98% confidence my original code with function names and debug info is unchanged.

Now GPT 4 is putting subtle typos back into the code i gave it.

Examples

console.error('Failed to create new group:', error);

became:

console.error('Failed to create new professor:', error);

and

const resourceGroups = await getExistingResourceGroups() ?? [];

became

const resourceGroups = await getExistingResourceView() ?? [];

and

} else if(line['Type'] === 'System') {

became

} else if(line['Tracer Type'] === 'System') {

Frustrating to say the least… but still hopeful that the quality goes back to what it used to be.

2 Likes
4

Exactly, your examples are exactly the kind of thing I am seeing. It so odd, but the typos are so subtle you will not find them until you start running the code.

I am going to take the suggestion to revert back to the previous GPT4 on the API with 8k tokens, the one without vision. This problem started with the increase of the context and adding vision to the normal GPT4.

1 Like
5

On the API I can revert back to the older model.

1 Like
6

I just want to add that GPT-4o is a degraded version of GPT-4, it is faster, but more stupid. The speed-up is not worth the accompanying stupidity.

7

did you try using temperature of 0.1 or even 0 on the API? i use it when generating code and i think it reduces the probability of generating wrong names.

8

Yes, I noticed the difference exactly when the CEO Sam Altman was fired. Shortly after, there were implementations, and as I have experience, ChatGPT started to show signs of overtraining. After the new user feedback method was implemented, it got worse. I could easily write with ChatGPT 3.5 turbo; it was really very easy and better than Google or Bing for finding information. However, with the CEO’s departure, some errors started to appear, which I believe are related to overtraining. Additionally, I noticed that it might be an attack on the user feedback system, training the chat to make certain errors.

For those of you having issues, the responses I read are quite useful. By using the base to utilize an OpenAI key, since you have premium access, you can set up a platform and link the model to it. It would be like having the model you mentioned, the 4 without the “o,” but the 4-o model has many advantages. It is truly an incredible model, very precise, and after the CEO’s departure, it has been the best so far.