Issues with the new default version of gpt-4o (October 2024)

API
1

Since the update of the new default version of gpt-4o (October 2024), I’ve been encountering some issues with its behavior compared to the previous version. Specifically, I’ve noticed the following:

  1. The model doesn’t seem to follow instructions as accurately as before.
  2. It often fails to call the function calls when it should.
  3. In some cases, it has returned a function call response as a regular message, instead of calling the function (it returned the function response but “stop” as finish reason instead of “function_call”)

I use the completions API, and due to these inconsistencies, I think I’ll revert to the older version, gpt-4o-2024-05-13, which behaves more predictably in my opinion. It’s a shame because the new version is considerably cheaper, but I can’t risk these inconsistencies.

Has anyone else experienced similar issues with the new version? I’d appreciate any insights or workarounds you’ve found.

Thanks in advance!

2

Do you know the exact date when they made the switch to “gpt-4o-2024-08-06” as the default version?

3

The first model to support structured output, gpt-4o-2024-08-06, is half the cost of gpt-4o-2024-05-13, but it seems that this model has many errors in function calls and classification mistakes in enumeration.

It seems unlikely that we can reduce the cost by half without losing anything, and the reduction in cost might cause a decrease in the accuracy of the model’s structured output.

Alternatively, it might simply be that the model’s adaptation to structured output is still insufficient, and the accuracy may improve over time.

4

Yes, it was October 2nd, 2024.