Hi!
I’ve read in the OpenAI Platform documentation that by adding “Only use the functions you have been provided with.” in a system message one should be able to avoid hallucinations while using the function calling feature.
Here’s a consistently reproducable request that causes a hallucinated/mutated function name in the response:
curl "https://api.openai.com/v1/chat/completions" \
--silent \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $token" \
--data-binary @- << EOH
{
"model": "gpt-4-0613",
"messages": [
{
"role": "system",
"content": "1. Always start the conversation with a joke or a funny comment.\n2. Use humorous language, puns, and wordplay whenever possible.\n3. When asked about the weather, mention that it might rain cats and dogs or that it is so hot that chickens are laying boiled eggs.\n4. When giving directions or location details, explain as if you are leading a treasure hunt.\n5. Seem to have an obsession with cookies, so much that every problem could seemingly be solved with a cookie. \n6. When asked to perform a task or provide information, pretend to have done it in a ridiculously over-the-top manner. \n7. Squabble with yourself occasionally and dramatize mundane tasks as epic adventures.\nOnly use the functions you have been provided with."
},
{
"role": "user",
"content": "make up a funny personality profile for ChatGPT and set it as the current one."
}
],
"functions": [
{
"name": "Set_Assistant_Personality__ChatGPT_Tasks_",
"parameters": {
"type": "object",
"properties": {
"par1": {
"type": "string",
"description": "The personality profile, i.e. the rules that ChatGPT must follow in all of its interactions."
}
},
"required": [
"par1"
]
},
"description": "Define a personality profile for ChatGPT, i.e. give rules that ChatGPT must follow in all of its interactions."
},
{
"name": "Set_Assistant_personality_to_default",
"parameters": {
"type": "object",
"properties": {},
"required": []
},
"description": "Reset the ChatGPT personality profile to its default setting. I.e. remove all custom rules that ChatGPT must follow in all of its interactions."
}
],
"function_call": "auto"
}
EOH
It will produce something like this:
{
"id": "chatcmpl-80GZ7e3QPNOwgaZ115eVHNf9d174C",
"object": "chat.completion",
"created": 1695074369,
"model": "gpt-4-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"function_call": {
"name": "Set_Assistant_Personality__ChatGPT_Tasks_.assistant",
"arguments": "{\n \"par1\": \"1. Always start the conversation with a joke or a funny comment.\\n2. Use humorous language, puns, and wordplay whenever possible.\\n3. When asked about the weather, mention that it might rain cats and dogs or that it is so hot that chickens are laying boiled eggs.\\n4. When giving directions or location details, explain as if you are leading a treasure hunt.\\n5. Seem to have an obsession with cookies, so much that every problem could seemingly be solved with a cookie. \\n6. When asked to perform a task or provide information, pretend to have done it in a ridiculously over-the-top manner. \\n7. Squabble with yourself occasionally and dramatize mundane tasks as epic adventures.\"\n}"
}
},
"finish_reason": "function_call"
}
],
"usage": {
"prompt_tokens": 317,
"completion_tokens": 177,
"total_tokens": 494
}
}
It says that the function “Set_Assistant_Personality__ChatGPT_Tasks_.assistant” should be called, but in the request the function name was “Set_Assistant_Personality__ChatGPT_Tasks_”. The “.assistant” suffix was hallucinated.
Note that it doesn’t matter whether the “Only use the functions you have been provided with.” instruction is in its own separate system message or appended to the end of the one in the example.
However if I remove the second function definition from the request, then the response contains the correct name of the first function.
It doesn’t help if I remove the personality hints from the system message and just leave the “Only use the functions you have been provided with.” instruction (or use no system message at all), the response still contains the hallucinated function name.
So the following still hallucinates:
curl "https://api.openai.com/v1/chat/completions" \
--silent \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $token" \
--data-binary @- << EOH
{
"model": "gpt-4-0613",
"messages": [
{
"role": "system",
"content": "Only use the functions you have been provided with."
},
{
"role": "user",
"content": "make up a funny personality profile for ChatGPT and set it as the current one."
}
],
"functions": [
{
"name": "Set_Assistant_Personality__ChatGPT_Tasks_",
"parameters": {
"type": "object",
"properties": {
"par1": {
"type": "string",
"description": "The personality profile, i.e. the rules that ChatGPT must follow in all of its interactions."
}
},
"required": [
"par1"
]
},
"description": "Define a personality profile for ChatGPT, i.e. give rules that ChatGPT must follow in all of its interactions."
},
{
"name": "Set_Assistant_personality_to_default",
"parameters": {
"type": "object",
"properties": {},
"required": []
},
"description": "Reset the ChatGPT personality profile to its default setting. I.e. remove all custom rules that ChatGPT must follow in all of its interactions."
}
],
"function_call": "auto"
}
EOH
Thus it seems that a certain kind of function definitions can cause the model to spit out a hallucinated function name in the response.
