Function Calling - How Does It Modify The Prompt?

API
1

From the documentation it’s not clear to me how a function schema modifies the prompt being sent into the completions API (and I’m not aware of a way to see the entire prompt being submitted, pls lmk if there is one).

One thing I wonder, for example, is if the “description” field for a function argument would render some of the information in my top-level prompt superflous.

Before using function calling, I would specify “few-shot” examples with json keys i’d like to extract, and show an example structure and then request json_mode output.

Something like:
“If the user asks for a specialist, respond like this”
{'user':user, 'specialist_type':specialist_type,request_time....}

And I would provide a couple examples of each type. The prompt would be fairly long and cover several different request types and examples.

With function calling and a provided schema, I assume I don’t need to do that anymore, since the schema defines the fields and structure I need. This would presumably make most of the prior prompt superflous.

Similarly with the descriptions field, I would assume this can take the place of much of what the prior prompt would be responsible for if additional detail was needed beyond just the function argument names. For example, maybe I don’t need to say “look for fields of ‘x’, ‘y’, and ‘z’ types and extract them” in the prompt, since the fields will be clear from the schema, and the description can provide additional context if needed.

All of this is speculation though, because I do not know how the function, it’s arguments, or the description field get injected into / combined with the actual prompt.

Previously, I could see precisely what the prompt was (excluding of course any OpenAI System Level prompts that aren’t disclosed). But using function calling with description fields, I can’t know how the prompt changes, so I can’t be sure how to tweak description fields / the prompt to make sure I’m not being redundant.

Any advice on how to approach this? Or in general how I can know what the prompt looks like when providing a function schema, with and without descriptions?

1 Like
2

Are you having trouble with API? (your Post is rather long and not clear, consider being more succint? That would help people to help you)

The provision of the tools details is not an explicit part of the prompt from our perspective but a separate set of function definitions under tools

The final internal prompt? Presumably that’s proprietary and secret. I wonder if anyone has “hacked” it? There are probably open source equivalents that might give us a clue.

3 Likes
3

Sorry it’s too long, I didn’t have time to write a shorter question :wink:

I’m trying to understand how providing a function schema alters the prompt, to avoid keeping redundant instructions in the prompt. The documentation is extremely sparse. For example, the optional “description” argument to a function simply says:

image
image1222×172 16 KB

Great. How does it accomplish that? Does it add the description to the prompt? Where does it add it? If it’s added to the prompt, then I don’t need to have something like “extract any dates in xyz format”, because the description could already handle that. It would be redundant to also include it in the prompt.

For Enum use, same question. Is it the equivalent of saying in the prompt “please pick one of the following values” in plain text? Again - something I would need to know to figure out what is important to keep in the prompt. Also it’s a bit misleading to even have an enum value if that’s what’s happening, since it will still return results that are not in the enum set sometimes.

Enum isn’t even mentioned in the docs, other than being shown in the example JSON call. It feels like a lot of this stuff is kind of an undocumented black box.

In general, it’s difficult to know how I should engineer the prompt if I don’t know how it’s being altered.

4

all of the current models are trained for function calling so by adding tools in your API call, they are already aware of its existence.

yes. no need to do that.

previously, you need to mention the functions in the system prompt/instructions. but nowadays, you don’t have to. if there are special circumstance that your functions need to be called (e.g. call funcA after funcB), then in such case, you can add it in the system prompt/instructions.

1 Like
5

Thanks. I realize they are trained for function calling, but what I don’t know is how the functions, their parameters, descriptions, etc are injected into the prompt. Why do I care? Because I don’t want to add extraneous information that has already been injected on my behalf.

6

here’s a reply from the openai staff about another question that might answer your question:

3 Likes
7

It doesn’t really answer the question, but thank you for the reference nonetheless. Even that answer displays yet another contradiction vs the docs.

The staffmember mentions that the System Prompt should be used to clarify when a function is to be called. The documents instead suggest using the explicit “tool_choice” argument to force function calls:

image
image1826×614 89.9 KB

In any event, my question is more broadly about how the presence of functions, descriptions, etc. effects the prompt, which as far as I can tell nobody knows.

8

Modification of the system prompt is optional, but I have found it helpful to add something like “You have tools available to you that you can call in the form of functions”. However I suspect the internal prompts cover this already.

You can use the system prompt to emphasise specific functions, but the LLM is quite capable of choosing a suitable function based on other context together with your function definitions in tools.

If you are using the Chat Completions API you must include the tools information on every call, nothing is stored remotely at the LLM.

Each call must have all the appropriate context included, including the tools information.

9

When you provided tools/functions in completions API, all the info from the tools definition (including description, param types etc.) is fed to the model as part of the prompt. That said, you don’t need to repeat the same information in the prompt. The models were trained to understand the meaning of function schemas.

In terms of customizing the model’s function calling behavior, the tool_choice param allows you to force the model to call (or never call) functions. But if you want the model to decide whether or not to call a function based on the context, the system prompt is the best place for that context, as opposed to inside function schema.

For example, if you are building an e-commerce agent for customer support which supports product returns via calling return_product(item_id) function, and you want to make sure to only return products the user have actually bought. In this case, I’d suggest you provid a list of products the current user has bought (e.g. A, B, C) inside the system prompt, so when the user asks “return product D”, the model would likely respond with “You haven’t bought product D, and we can only return products you’ve bought – A, B, and C.”. An alternative is to dynamically define the schema of the item_id param of the return_product function as an enum of value [A, B, C]. With this setup, when user asked to “return product D”, empirically, the model is more likely (compared to the system prompt approach) to call return_product with a hallucinated param value.

3 Likes
10

Hmmm. That’s not how I’d recommend doing it. Surely it’s most appropriate to respond to the LLM accordingly as part of the answer from the function?

11

I don’t think I fully understand your comment. Can you elaborate?

12

I’m using your API for a Production Chatbot.

That Chatbot is relatively sophisticated now. All the functional outputs are passed to the LLM via answers and that includes error conditions like the one you mention (as necessary). I do not need to modify the system prompt.

If that is sub-optimal I’m all ears.

13

That’s exactly what I did in a project where I had to implement the NASA Open API for getting imagery.

For context: You can’t simply query the API for imagery on a particular date. The process requires two steps:

  1. Request data from the Earth Assets API to find the closest satellite pass for a given location and date.
  2. Use that data to query the Earth Imagery API with the exact date-time and location.

Initially, when users asked for imagery, the model would use their exact date-time to call the Imagery API directly, resulting in errors.

I solved this by adding instructions to the system message, directing it to first query the Earth Assets API for the closest pass data, then use that information to fetch the correct imagery.

14

Thanks… that’s helpful. I assume you mean that using the enum it’s less likely to hallucinate?

In terms of using the system prompt as a way to hep the model decide when to call the function, I thought the “description” field could be used as an alternate. This is exactly why I’m asking the question, because it’s not clear to me the tradeoffs in each approach.

I think it would be helpful to have the docs provide an example of how a function schema makes it into the prompt. If it’s proprietary, you could show where it’s added and have placeholder data for the sensitive bits, e.g.:

<sys prompt>
{tools:[funciton..]}
<user prompt>
<end prompt>
15

According to my understanding, when using function calling, the tool spec is passed to the model in the system message in a predefined template, that the model was trained on, likely consisting of special tokens to demarcate the beginning and end of the tool spec. It is not yet revealed what those tokens exactly are.

1 Like
16

This is a good succinct example. I wonder if specifying that constraint in the “descriptions” field would have resulted in a similar outcome. Not suggesting one way is better than the other, just curious how they both work.

1 Like
17

It can be done, but if, say from the example, there are functions apart from the two that can be used independently, then it has chances of interfering with the model’s perception of whether it should call the function that says “call me first and use my data for other function calls” in its definition.

18

If there are more than two functions, presumably the description field could specify which is to be called first, instead of just saying “me”. E.g. “this function should be called after Request Data, which must first return a given location and date in order for us to proceed”

Again not suggesting one way or the other, just trying to understand the tradeoffs, and be aware of when description fields / function arg types are “overlapping” system prompt.

19

I was not suggesting modifying system prompt after function calls. In my example, I was describing how to set up the initial system prompt, i.e. fetching user bought products from your internal system before starting the chat session. For new information returned by function calls suggested by the model, passing it back as answers to LLM is the right way.

3 Likes
20

A k, got it. I was imagining this as part of an existing ongoing dynamic session. thanks for clarification.