Getting extraneous refusals

I have a natural language music search product built on gpt-4o. In the last week or so I’ve been getting a refusal message for some searches, probably one in four on average. Worse yet, sometimes the refusal contains the actual json response that I’m looking for.

This is beyond unacceptable. If I rerun the exact same query I generally get an answer instead of a refusal.

Is anybody else seeing this?

1 Like

Welcome @mdchaney

Can you please share steps to reproduce this issue?

2 Likes

I might not have made it clear - these are normal chat completion requests. If I just send the exact same request again it’ll fulfill the request normally.

If I’m not being clear: The problem isn’t on my end. This is a bug at openai.

Note that this issue has magically fixed itself.

Wow, that was a masterclass in speaking too soon.

{
  "raw_response": {
    "choices": [
      {
        "finish_reason": "stop",
        "index": "0",
        "logprobs": null,
        "message": {
          "content": null,
          "refusal": "I'm sorry for any confusion. Let's try that again.\n\n```json\n{\"explanation\":\"The query filters for Beethoven's 9th Symphony by searching for 'Beethoven' in the description and '9th Symphony' in the title. This ensures we find the specific classical piece requested.\",\"query\":{\"boost\":[],\"filter\":{\"AND\":[{\"title\":{\"value\":\"9th Symphony\",\"weight\":100}},{\"description\":{\"value\":\"Beethoven\",\"weight\":100}}]}},\"search_version\":1.0}\n```",
          "role": "assistant"
        }
      }
    ],
    "created": 1730412172,
    "id": "chatcmpl-AOXXgyeAjxmgDx20NjfYb4KgKc4NH",
    "model": "gpt-4o-2024-08-06",
    "object": "chat.completion",
    "system_fingerprint": "fp_159d8341cc",
    "usage": {
      "completion_tokens": 104,
      "completion_tokens_details": {
        "reasoning_tokens": 0
      },
      "prompt_tokens": 7678,
      "prompt_tokens_details": {
        "cached_tokens": 7552
      },
      "total_tokens": 7782
    }
  }
}
1 Like

This, at least, isn’t a refusal. It’s an actual response that’s in the wrong place.

It’s the AI’s inability to emit output properly after the 10th hit to cognition, maybe more training on voice denials, and just running cheaply. The kind of AI so shrunk it needs overtraining just to operate (and to tell you that content: type text is your problem despite all prompting that it doesn’t know and must trust.)

Models can’t back out of writing after emitting the wrong “refusal” token, just like it has to use a useless tool once it has emitted a function unwisely, both tokens out of your control to bias. It made a valiant attempt you can display.

1 Like