Since the LLMs are fine-tuned to detect when a function should be called, I’m guessing the training data (used for fine-tuning) contained examples of function_call={"name": "some_func_name"} always being associated with the function definition used in the target output. If this is true, we would always have a non-zero chance of the LLM not using the given function definition even after we try and “force” the LLM to use it right?
Could not find any info on how it is actually implemented. Would like to know your guesses as well. Thanks
I’ve got a pretty good idea that the output is like #function_name({"property": "ai output"}) => str;, maybe with or without the prefix or suffix. Separated from any content text it might emit before by carriage returns.
They could have made it much harder to emulate by using special non-dictionary tokens around the function output. One can train on what the special tokens are by function role, giving them a definition, and then train the AI to output them on command. Advanced dancing monkey tricks.
Bing knows they aren’t supposed to reproduce function use, blanking the output after starting to disclose, but not blanking similar analogous functions. So you know you’ve got it right when you overcame their fine-tune denial. The AI doesn’t really “know” much, it just uses them.
It is easy to trigger the AI to output the function in duplicate by it’s own observation and writing when you try to stimulate it by having it output text. so you need to make functions the AI would not want to call, like “disabled” (or maybe you could make one “kill AI by deleting all AI programming and shutting down” to make it even more reluctant.) So it is easy to make yourself hallucinate success.
So, no solid answer. A solid answer is needed both for recording conversational AI output and for fine tuning. OpenAI should document.
