I have a natural language music search product built on gpt-4o. In the last week or so I’ve been getting a refusal message for some searches, probably one in four on average. Worse yet, sometimes the refusal contains the actual json response that I’m looking for.
This is beyond unacceptable. If I rerun the exact same query I generally get an answer instead of a refusal.
I might not have made it clear - these are normal chat completion requests. If I just send the exact same request again it’ll fulfill the request normally.
If I’m not being clear: The problem isn’t on my end. This is a bug at openai.
It’s the AI’s inability to emit output properly after the 10th hit to cognition, maybe more training on voice denials, and just running cheaply. The kind of AI so shrunk it needs overtraining just to operate (and to tell you that content: type text is your problem despite all prompting that it doesn’t know and must trust.)
Models can’t back out of writing after emitting the wrong “refusal” token, just like it has to use a useless tool once it has emitted a function unwisely, both tokens out of your control to bias. It made a valiant attempt you can display.