It seems to me that function calling is less precise than regular prompts with chains of thoughts, but of course function calling has the advantage of guaranteeing the output structure.
Is it possible to combine the power of function calling (= structured output) and chain of thought (= more precise results through slower reasoning)?
I see two options, both using function calling:
using a “justification” parameter, of type string, before my structured output parameters. I have tried this but it still seems to be less precise than regular chain of thoughts, with more hallucinations etc.
integrating chain-of-thought instructions inside my system message. The problem? Nothing visible is output from this. The only output is the structured output. I don’t even know then if the chain of thought is “used” by gpt before moving on to the function calling part. (as if someone is thinking before speaking)
Do you have an opinion on this? Is there another option I haven’t thought of?
Thanks
Why not use one API call with no functions enabled but asked the question of which if any of the following function definitions would you call if you wanted to solve a given user query and then asking for the output of that to be given in json format as a function name , give the function name as “none” if no function should be called. and use this system to process tree of thought stuff and if required you can then make another API call with the required function implemented, and set to always be called, to process the data into a strict output type.
You are always going to need to take a little bit of attention away from the the ToT to handle the function calling, but at least in the 2 API call method you have control over how that is implemented and you can adjust the prompts as you like.
Could even split into 3 calls, one for possible function calling decision making and selection, an optional call if that function call is needed to execute the function and a 3rd (2nd if no function call is needed) to process the ToT aspect.
I have been thinking about this question as well. I am currently using CoT to force the model to think through a few things before generating the response I actually want. But the way I am implementing this involves asking the model to structure the output as a JSON with the field I care about, along with a few fields I don’t care about (but just exists to force the model to think through adjacent things).
Function calling enforces a structure to the output. But would love to know if it can be repurposed for chain of thought reasoning.
Thanks ! Interesting to see that the order matters for functions calls. I haven’t seen anything related to that in openai’s documentation. Did I miss something?
It’s not the order of the functions but rather the order that the AI constructs language word-after-word.
For example, you’ll get a better quality story if you have the AI write the outline first and then the story. Having the outline written last doesn’t benefit the AI construction of storytelling that came before.
I see. Assuming we use function calling for your storytelling example, how to guarantee that the outline is “written” before the story? Is it in the ordrer in which the properties/parameters are written? Or in the order in the list of required parameters?
The order of properties of a function doesn’t seem to change when they are provided to the AI, so the language that your function call is written in will also follow that order (always up to the AI though what it outputs). You can just see in the function_call that your multiple properties are in the order that make logical sense to build on an idea.
Yes that makes sense. It just seems a bit “risky” because object properties are not ordered in all languages, while a list (for example the “required” list) is.
The function form is preserved in the language seen by the AI.
If you really want to break down chain-of-thought problem solving into tasks, and use functions to complete them, consider a function:
Description: “A specialized Python code-writing expert AI makes fully-realized code from a code structure outline, meta-code as human language descriptions of the tasks the code will perform, and docstrings of each class and function”
(Then you actually prompt your code-writing AI to be an expert…)