Hey!
I’ve been dealing with a situation, where GPT decides to generate a random function parameter, in order to execute a certain function. (At the moment, using GPT 4 0125 model)
Here is a mock of one of the functions, to have it as an example:
@meta(
name="sell_domain",
description="This function gives information, on how to sell a specific user's domain. ",
parameters={
"type": "object",
"properties": {
"domain_name": {
"type": "string",
"description": "user's given domain name. This domain must be taken from user's input.",
}
},
"required": ["domain_name"],
},
)
As you may see, this function expects to receive a domain_url from the user’s input.
Now let’s imagine, that in the function, I am doing certain checks, to see, where the domain is purchased from, in order to personalize the steps, so they would match exactly what the user needs to do, rather than giving general information.
Usually (70% of the cases), GPT doesn’t call this function, without knowing the user’s domain name, although, in the rest of the cases, GPT decides to give a random domain name, because the user hasn’t mentioned any in his message. GPT doesn’t try to clarify with the user, and instead, provides a parameter like: “example_com”, “domainexample_org”, “non-existing-domain_net”…, etc. (had to change dots to underscores, since this form doesn’t allow links) Basically, a domain name, as an example, which in general, is still a valid domain, that cannot be re-verified by some regex.
For example, the user’s input could be: “Hey, I’d like to sell my domain”
Since the domain is still valid, my checks are going through, I locate where the domain is purchased, and I provide steps for that exact domain. GPT takes that response as truth, and simply provides these steps to the user, which leads to a bad experience because the steps are far from being correct since the user has the domain purchased somewhere else.
So far, I’ve been trying to deal with this issue in these ways:
- Better prompt - one of the main things, that helps to reduce this random domain generation, but still, not even close to it being perfect, GPT still decides to call this function, without having required slots filled by the user. No matter how extensively I say in system/function descriptions, that you must clarify with the user about his domain name, you cannot generate random domain names…
- Removing the “required” slot from the meta, so GPT instead gives an empty string/doesn’t provide a domain at all, as a parameter. Then simply defaulting the domain_name param in the function to None, and doing a simple if check, returning the message, that it’s required to have a domain name from the user’s input.
- Having a list of most commonly randomly generated domains by GPT, validating against it, and returning if GPT generated a domain, I expect to be random. This is a workaround, rather than a solution, as GPT still generates some sort of random domain, that I haven’t added to this list.
Even after these multiple workarounds, I still face an issue, where GPT generates me most random domain ever, causing incorrect answers generated by the function. It feels like GPT simply wants to use the function, without expecting to get the answer that will solve the user’s issue.
Maybe you have faced anything similar in any context, and have any solutions for it? Maybe you have any bullet-proof prompts that work for such cases? Maybe there are some other solutions, that are similar to my mentioned one about removing the required field?
Any ideas are appreciated, as this has been a headache for a long time.