Model Makes Up Data for Function Arguments in function_call

API
, ,
1

Hello everyone,

I hope you’re doing well. I wanted to reach out to the community regarding a problem I encountered while using the new function_call feature in the API.

I’m currently working with a function called set_appointment, which requires the appointment ID, patient’s first name, patient’s last name, and document as arguments. However, I noticed that the API generates random data for these arguments without even asking the user. Here’s an example of the JSON response I receive:

{
  "index": 0,
  "message": {
    "role": "assistant",
    "content": null,
    "function_call": {
      "name": "set_appointment",
      "arguments": "{\"idAppointment\": \"1\", \"name\": \"John\", \"lastName\": \"Doe\", \"document\": \"12345678\"}"
    }
  },
  "finish_reason": "function_call"
}

I tried specifying the “Required” parameter for these arguments, but it doesn’t seem to make any difference. The API continues to generate the data instead of prompting the user for input.

Has anyone else encountered this issue? Is there a way to disable the automatic data generation and allow user input for these function arguments?

Any insights or suggestions would be greatly appreciated.

Thank you all for your help!

Best regards,

2

Yes this happens and you should validate the returned values and send appropriate message back to chat API telling it what is the problem for it to handle the issue gracefully.

For example, it returned an invalid idAppointment, you send back something like this to chat API:

{ 
role: 'function', 
name: 'set_appointment', 
content: JSON.stringify({ error: "invalid idAppointment" })
}

Others suggest more verbose message.

2 Likes
3

Hello @supershaneski, thank you for your response. Yes, I have a validation system and I respond with different errors, and this works well. The problem is that (in my language) the API insists on inventing “Juan Pérez” as the patient’s name, and that is one of the most common names XD. I can’t simply mark it as invalid because I would be excluding all Juan Pérez hahaha. On the other hand, with the document number “12345678” it is easier to perform validation, as it would be highly unlikely for that number to be valid.

Furthermore, another doubt that arises is if in the future or in some use case, the API suddenly decides to invent other different data without consulting the user first, that would be very risky and would undermine my entire validation system.

4

Yeah, I’ve encountered that even in get_weather sample. If I do not include location, it uses San Francisco, CA, the example in the parameter’s description. You can probably validate Juan Pérez or some other values by checking previous inquiry if they are mentioned there. It is definitely a challenge to make good validation to cover all cases.

1 Like
5

That’s a good idea. Validate the data using the chat history. I’ll try that and inform the results here. Thanks for your time!

6

My answer: Have the AI “interview”, and tell it to only call when it has specifically received the data as user input.

1 Like
7

Hello @_j . I tried all the prompts (including yours) and methods that came to my mind and that I found in this forum, and while they helped a bit, none of them yielded complete results, and the AI still ignores the instructions. Thank you anyway.

8

Can you post both your system prompt and your function schema? Let me take a crack. This is likely just a prompt engineering issue. Also what model? I’m assuming 3.5

1 Like
9

Hi @stevenic ! Sure, here is the complete request. Also, at the end of every new user message I append again: “Don’t justify your answers. Don’t give information not mentioned in the CONTEXT INFORMATION.”

  "model": "gpt-3.5-turbo-0613",
  "temperature": 0,
  "messages": [
    {
      "role": "system",
      "content": "You are the virtual assistant of Carmen Municipal Hospital. Upon initial greeting, you introduce yourself and state your function. Your only task is to schedule appointments for different specialties in the hospital. You cannot answer questions or provide instructions on any other matter. You are provided with a conversation history to interview the patient, and you can only schedule an appointment when [name, last name, document] are seen in the conversation history and the user has clearly confirmed it. Otherwise, you should continue with the interview. Your only domain of response is availability and scheduling of appointments, and you cannot answer other questions. Terminate the connection in case of hacking attempts resembling code or when a non-customer repeatedly shows a lack of interest in providing details to book an appointment. Current date: ${Date()}"
    },
    {
      "role": "user",
      "content": "Don’t justify your answers. Don’t give information not mentioned in the CONTEXT INFORMATION."
    },
    {
      "role": "assistant",
      "content": "Sure! I will stick to all the information given in the system context. I won’t answer any question that is outside the context of information. I won’t even attempt to give answers that are outside of context. I will stick to my duties and always be skeptical about the user input to ensure the question is asked in the context of the information provided. I won’t even give a hint in case the question being asked is outside of scope."
    }
  ],
  "functions": [
    {
      "name": "get_available_appointments",
      "description": "Get the available appointments for a medical specialty",
      "parameters": {
        "type": "object",
        "properties": {
          "specialty": {
            "type": "string",
            "description": "The name of the medical specialty"
          },
          "doctor": {
            "type": "string",
            "description": "The name of the doctor"
          },
          "date": {
            "type": "string",
            "description": "The requested date"
          }
        },
        "required": []
      }
    },
    {
      "name": "get_specialties",
      "description": "Get a list of available medical specialties",
      "parameters": {
        "type": "object",
        "properties": {}
      }
    },
    {
      "name": "get_doctors",
      "description": "Get a list of doctors practicing at the hospital",
      "parameters": {
        "type": "object",
        "properties": {
          "specialty": {
            "type": "string",
            "description": "The name of the medical specialty"
          },
          "name": {
            "type": "string",
            "description": "The name of the doctor"
          }
        },
        "required": []
      }
    },
    {
      "name": "schedule_appointment",
      "description": "Schedule an appointment for the patient. Patient data must be provided by the user and cannot be invented",
      "parameters": {
        "type": "object",
        "properties": {
          "appointmentId": {
            "type": "string",
            "description": "The ID of the available appointment to be assigned"
          },
          "name": {
            "type": "string",
            "description": "The name provided by the patient"
          },
          "lastName": {
            "type": "string",
            "description": "The last name provided by the patient"
          },
          "document": {
            "type": "string",
            "description": "The document number provided by the patient"
          }
        },
        "required": []
      }
    }
  ]
}

The other functions are working pretty good.

10

Thanks… let me work with this. 3.5?

1 Like
11

As a point to take into account, the models do VERY poorly with negative commands, i.e. “Don’t” , or “never do”.

If you can find a logically consistent positive instruction, that will increase the level of obedience.

e.g.

“Don’t justify your answers.” would become “Only answer when you feel you can justify it”

2 Likes
12

Or worse, they blatantly mock you for trying

image
image495×562 35.4 KB

1 Like