I am following the OpenAI cookbook to build an Assistant that operates on a meeting summary and actions upon that summary. There are somewhat complex logic to figure out the set of functions to call, and ~20 custom function calls as tools. Even though I pass in some of that logic into the run instructions, the model still sometimes doesn’t follow those instructions.
How can I get the model to follow those instructions, or in general improve reasoning ability in terms of correctly identifying the sequence of functions to call? What are the current best practices there?
Very detailed Assistant instructions that specify under which circumstances and in which sequence functions to call is a good starting point.
Likewise, the description of your function calls also benefits from specificity.
If you are willing to share an extract of your current Assistant instructions, we could take a deeper look at potential improvement opportunities.
There’s also the option to upload examples via the file upload of how an Assistant should operate in response to user query in your specific context and then reference the file in your instructions. This may not always work 100% but can yield performance improvements.
I guess you had a look at the documentation already (https://platform.openai.com/docs/guides/function-calling#best-practices-for-defining-functions)
I don’t have a definite answer but I can quickly describe what I’ve done so far which helped to some extent. It’s probably not best practices or what but it helped, even though I’m still not 100% happy with my assistant behavior when I increase the number of functions.
I avoided overlapping keywords in the function descriptions so that the llm doesn’t “hesitate” between them.
Sometimes, I had to explicitly state in the instructions to exclude specific functions under some circumstances. (e.g., 'you can interpret images (use function interpet_image but DO NOT use get_sandbox_image)).
I built a kind of “prompt optimizer” script which sends my assistant instructions and functions description to the LLM with instructions asking for optimization, for highlighting potential confusion between function descriptions, suggesting more specific descriptions etc… It helped initially, especially with the formatting of my instructions, but on the longer run it’s far from good enough and I’m barely using it anymore.
If possible, don’t systematically enable all your functions. e.g., my assistant is used by both office and scientific people. The office people don’t care about retrieving compounds or lab journals. So for office people I’m loading the assistant without any access to scientific functions (obvious) but for the scientists, I ended up adding a kind of “scientific mode” toggle which would let them decide if they just want a standard assistant for office like stuff or if they also need to access their own APIs. Not so happy about having the user to decide and I’ll see on the longer term if it works but it prevents some issues where the LLM systematically called scientific APIs while it was supposed to use the more basic things to answer the user questions.
What I’m considering/haven’t done yet:
Fine-tuning (either to reduce the number of functions I actually need, or to orientate the decision making).
have an extra layer of “answer validation” before giving the assistant response to the user and if not appropriate, go for another round before replying
…
Does anyone have some good references/tutorials/webinars about assistant function calling optimization?
Most of the things I went through are rather about about how to do function calling on a basic level or people selling their product that does orchestration or so… :(. Will keep looking for more and let you know if I find any but I would have expected plenty of these as, imo, there is still a challenge about instructions and function descriptions if you want your assistant with many functions to perform efficiently. (or people are not interested anymore because now it’s all about multi agent and splitting the functions across ).
).