How do we submit evals for function calls?

[sam.saffron]

I am noticing quite a few cases where evals for function calls can help

1. While testing I made a searchbot and asked it to search for “404 in user profiles.” no matter what I did I only got “profiles404 user” in my function args.

2. While testing I noticed that no matter what I do I can not get the model to run multiple functions, “search for car and then bar, what is common?”

Reading through eval source code I can see that the completions only look at the returned message:

openai/evals/blob/c2587c69a2f330282d3ba76eceaf580ee03fa67a/evals/completion_fns/openai.py#L28-L35

I tried searching through the repo for function_call and can not find anything.

Can OpenAI add a sample eval for function calls, then we can all together help hone the functionality!

[dani-mp]

I faced this problem myself today. Is it even possible to include the functions in the first place?

[vladin]

Hey, any progress on that? In fact would be nice extending the eval example to include “functions calling eval” as well as LLM eval. Basically end-to-end eval of the graph.
It would also be nice to allow a quick adding of test case to eval set right from the dashboard, providing its ground truth.