Hi, I had the same problem, and here’s how I solved it:
Unfortunately, I can’t share the my blog post link, but here’s a brief explanation of the method:
Start with a Single Tool: We begin by using a special tool called the “tool descriptor.” This tool takes one parameter, a list named neededTools, which specifies the tools that might be needed. You list all the available tools here.
Requesting Specific Tools: If the AI determines it needs certain tools to complete its task, it requests them through the “tool descriptor” by specifying which tools it needs from the neededTools list.
Providing the Requested Tools: Once the AI requests specific tools, we then supply these requested tools to the AI.
AI Uses the Tools: Now that the AI has the tools it specifically asked for, it can go ahead and process the request, using the tools as needed to come up with a final answer. Occasionally, during this process, the AI might realize it needs an additional tool it didn’t request initially. If that happens, the process starts over, and we provide the newly requested tool.
Github: /MaurerKrisztian/two-step-llm-tool-call
1 Like
Sorry for the late reply here… I would have suggested the use of embeddings and similarity search as well. Since you’re already using that approach and seeing good results, let me give you some ideas for how to further improve things.
Let’s take an example… Say you have a query of “when does Jane turn 21?”. The issue is the query may have very weak semantic similarity with the function “get_contact” even if the description contains the word “birthday”. The issue is the “intent”, lookup Jane’s birthday, isn’t captured by the embeddings generated for the query.
As you noted, semantic search often will return the best function to use in the top 3 or 4 results but there are cases where it won’t. And the more functions you have the greater the likelihood that you won’t return the correct function in the top 5 - 10 results.
When I worked at Microsoft I helped the Office Copilot team solve this exact problem. They had something like 600 possible functions in their library and they were struggling to select the correct functions to include in the prompt using “query → description-> function” mappings alone.
The solution I came up with for the Copilot team was to add to their Vector DB of functions additional “query → query → function” mappings. Instead of a single entry for each function in your VectorDB for you have many entries. There’s 1 entry that includes the embeddings for the functions description and then many entries for queries that should predict the use of the function. These added embeddings re-enforce the ability to semantically predict the likelihood of a function.
At query time you go through the top n results (say 100) and then compute a weighted average of the scores for each function. This lets you essentially re-sort the list of functions returned from the vector db and with enough “query-> query → function” mappings you should see that the best function to use for a query is always in the top 2-3 functions returned regardless of the number of functions in your library.
If you have a feedback mechanism (thumbs up/down) you can automate everything by automatically adding mappings for any thumbs up mappings to your vector db.
2 Likes
