I’ve been playing around with the new functions, and it’s alright with dealing with input → format conversion, but it seems to struggle with inference.
For example, I’m trying to make an image search function, I have it declared like:
{
"name": "search_image",
"description": "Do image search using inferred keywords.",
"parameters": {
"type": "object",
"properties": {
"keywords": {
"type": "string",
"description": "Query answering the user request."
}
}
}
}
The issue I am facing is that when I provide a query like “give me a picture of that animal with the long neck”, the function doesn’t infer the specific keyword like “giraffe”. Instead, it simply returns “search_image{“keywords”: “animal with long neck”}”.
(which does bring up giraffes, but that’s thanks to search engines)
And it’s the same both on gpt-3.5 and gpt-4, although with higher temperatures, gpt-4 sometimes gets it right, but then it more often doesn’t call the function at all.
So what I want to know if I’m using it right and if this can be fixed via better description, or is it something that the model is weak at currently?
(And I know I can fix it with another regular request before the function call, but then that defeats the purpose of them.)
Perhaps the “Do image search using inferred keywords” isn’t exactly what you’d want. “Keywords” tells it to do an extraction of those words, while “image search” means some intelligence on the part of the search function. And the actual “command” part of “Query answering the user request.” is quite ambiguous. Query can be a verb or noun.
get_image: returns the closest matching image from a library using the precise details the AI must provide.
{
“name”: “search_image”,
“description”: “Perform an image search by interpreting and inferring context from provided keywords.”,
“parameters”: {
“type”: “object”,
“properties”: {
“keywords”: {
“type”: “string”,
“description”: “User-provided keywords that will be used to infer and identify the images they are seeking.”
}
}
}
}
Thanks for the suggestion. But no luck with this nor @_j’s descriptions.
Also tried phrasing it as a query function, keyword inference/extraction function, and splitting it up into two parameters, the original message and the query.
But it still would rather insert the request directly. Also noticed that the “required” property is weakly enforced and also needs some description finagling to get it to include parameters.
It feels like they’re using a weaker model (ada?) under the hood for function calling as I had similar troubles with it back then too, but even simulating the request using prompts it seems to struggle with it.
I’m tuning them while working on it, but right now I have this little testing script.
import openai
openai.api_key = "<KEY>"
func = [
{
"name": "search_image",
"description": "Do an image search based on a message context.",
"parameters": {
"type": "object",
"properties": {
"keywords": {
"type": "string",
"description": "COMMA SEPARATED Semantic keywords based on the message."
}
}
},
"required": ["keywords"]
}
]
def respond(message):
sysmsg = {"role":"system", "content":"you turn user requests into searchable keywords based on context"}
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo-0613",
#model="gpt-4-0613",
messages=[sysmsg, {"role":"user", "content":message}],
functions=func,
temperature=0.4
)
txt = ""
if response.choices[0].finish_reason != "function_call":
txt = response.choices[0].message.content
else:
txt = str(response.choices[0].message.function_call["arguments"])
return txt
print(respond("show me a picture of that animal with the long neck"))
Pretty much any non-specific prompt is passed to parameters verbatim. For example "what's that famous tower in france called? show me a picture of that" or "find me a picture of an animal that goes 'meow'".
I think the key to this may be to trick it with another function and then just use that one as the image search. But for now my imagination is lacking.
That’s it.
Well, for the actual project I have a longer one, that’s about short and direct responses. Which does influence the regular responses, but no effect on the functions.
For now I’m trying to make the test work to see if it’s even possible to achieve what I need.
You might try putting something like " “description”: “Perform an image search by interpreting and inferring context from provided keywords.”," in the general system prompt too… or something similar…
Oh, that actually gave a better result. Still not consistent even on GPT-4, but definitely a step in the right direction. Will try tuning it more. Thanks.
Though it will be a challenge integrating it with other functions without affecting the regular prompt.
Hope that OpenAI increases the description weight, as for now it’s barely a suggestion.
I was going to suggest the same thing. The instructions you give in the system message are just as important as well defined descriptions for your function usage and parameters. I get really good results by just telling it what I want it to do as a list of steps…
Ask the user what kind of image to search for.
Use the image_search function to find a list of images.
Present the results to the user as a bulleted list of links.
You are right, I think using the open API specs directly is not the way to do it. The open API specs need to be tweaked. I am not sure if any openai developers are reading this, but if they are then they should rethink how they are approaching this.