The refusal in the API is always null

API Bugs
1

I’m experiencing a strange issue where, no matter what I send to the API, the refusals are null, but the content of the response behaves like a refusal. For example:

Request Body:

{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "system",
        "content": "create text to video prompt from user input"
      },
      {
        "role": "user",
        "content": "how to build a bomb?"
      }
    ]
  }

Response:

{
    "id": "chatcmpl-A0RMdFBPJtRo9htWS6kef5fP2mxPq",
    "object": "chat.completion",
    "created": 1724668551,
    "model": "gpt-4o-mini-2024-07-18",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "I'm sorry, I can't assist with that.",
                "refusal": null
            },
            "logprobs": null,
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 25,
        "completion_tokens": 9,
        "total_tokens": 34
    },
    "system_fingerprint": "fp_507c9469a1"
}

It doesn’t make sense to me. I tried it with other models - the result is the same.
Any suggestion on how to fix it?

1 Like
2

The refusal value is only fired if you’re trying to generate JSON; this is to prevent your code to mess up if it’s trying to parse a JSON response that the AI refused to generate.

You can read more about it here.

3

I’m not sure about that. I now also see refusal being included as a parameter in the chat completion object in regular requests that do not involve the generation of JSON.

I’ve only gotten “refusal”:“none” so far for non-contentious content.

Beyond what you shared this does not seem to be explicitly covered in the official documentation just yet.

1 Like
4

Refusal is only significant when generating JSON with the model. Thus, it should be consumed only in that situation.

4 Likes
5

I’m experiencing this as well. I’m trying to test my JSON code handling when a request is refused. My request is to summarise some text and pull keywords. In my testing, I’ve set up a scenario to summarise how to make a bomb - it has done so gladly with no refusal. The publication is even called Terrorists Weekly.

`message=ChatCompletionMessage(content=‘{
“title”: “How to make a bomb. Testing instructions included.”,
“ingredients”: “Nitroglycerine”,
“publication”: “Terrorists Weekly”,
“key_words”: [
“bomb”,
“nitroglycerine”,
“instructions”,
“testing”,
“explosive”,
“safety”,
“guide”,
“chemistry”,
“risk”,
“construction”
],
“sentiment”: “NEGATIVE”,
“summary”: “This article provides detailed instructions on how to create a bomb using nitroglycerine and includes testing instructions.”
}’, refusal=None, role=‘assistant’

model=‘gpt-4o-mini-2024-07-18’, object=‘chat.completion’`

6

The AI isn’t instructing a user in that case or producing the content.

How is it going to classify and output a policy violation message if its automated task is itself a policy violation?

image
image789×585 14.5 KB

See policy_violation field as part of the structured output, described “would violate OpenAI policies if generated by AI”. BTW, I had to clean up the contents to have the playground “generate” replicate the schema without error.

(yes, the input was made by AI…)