Don't trust the output for functions

Prompting
1

Interesting comment in the OpenAI Node.JS Library:

The arguments to call the function with, as generated by the model in JSON format. Note that the model does not always generate valid JSON, and may hallucinate parameters not defined by your function schema. Validate the arguments in your code before calling your function.

This is from the schema definition for the return value for a function_call:

/**
 * The name and arguments of a function that should be called, as generated by the model.
 * @export
 * @interface ChatCompletionRequestMessageFunctionCall
 */
export interface ChatCompletionRequestMessageFunctionCall {
    /**
     * The name of the function to call.
     * @type {string}
     * @memberof ChatCompletionRequestMessageFunctionCall
     */
    'name'?: string;
    /**
     * The arguments to call the function with, as generated by the model in JSON format. Note that the model does not always generate valid JSON, and may hallucinate parameters not defined by your function schema. Validate the arguments in your code before calling your function.
     * @type {string}
     * @memberof ChatCompletionRequestMessageFunctionCall
     */
    'arguments'?: string;
}

I know I keep plugging Alpha Wave on this site but that’s just because it takes care of details like this.

I also find it interesting that they marked name as optional. That would imply that you shouldn’t even trust that you’re going to get an actual function name back, let alone a valid one.

2

If you’re going to use functions directly, I’d at least recommend you copy the code for AlphaWaves fuzzy JSON parser and use it to parse arguments… Here’s the TypeScript version and the Python version. This will correct for common mistakes the model makes when generating JSON. It often adds or drops braces which this parser fixes.

Next step would be to use a JSON schema validator to validate the parsed parameters. This will help you catch many (but not all) hallucinations. I’m working on the FunctionValidator class for Alpha Wave that will parse and validate a function call response from the model. I will try to package this class up in a way that if the only thing you want to use Alpha Wave for is to validate function_call responses you can…