Describing Meaning Behind Data Retrieved from Function Call

[chandler]

I am making functions available to GPT-4o which result in API calls which fetch external data. Then, following the advised pattern, I’m feeding the response from the API back into GPT-4o to allow it to derive insights.

First Question: Where is the best place to inform GPT-4o about what data can be retrieved from a function?

I see two options here:

A) In the user / system message sent to GPT-4o (“You can request details about a given ______ which will be returned like _______ and mean _______.”)

or

B) In the tool definition using the “description” field, see OpenAI’s example below.

{
    "name": "get_delivery_date",
    "description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {
                "type": "string",
                "description": "The customer's order ID.",
            },
        },
        "required": ["order_id"],
        "additionalProperties": false,
    }
}

Second question: After GPT-4o has selected a function, and the data has been retrieved from the corresponding API, where is the best place to tell GPT-4o how to interpret that data?

I see only one option here: In the system / user prompt, which would have to be contextualized based on which function GPT-4o chose to invoke (and therefore which API output it will have to interpret). Is this correct?

Thank you very much (and apologies if this is ill-formed). I did try to search on this but wasn’t able to find the answer I needed.

[_j]

There is only one way to return the results of a tool call (functions):

1. append the assistant response that was emitted as a message
2. append the tool return text as a message.

Both of these require matching IDs, because there may be multiple functions emitted at once, and require the assistant + tool pair to both be placed in the proper form. You can follow the link below for a full demo, where here I quote just sending back the value (extending the python messages of the API call)

@_j

# Return values: what the tool_call with multiple functions gives
# rename xparams to params here also for the 2nd run
xparams['messages'].extend(
[
  {
    "role": "tool", "tool_call_id": "call_rygjilssMBx8JQGUgEo7QqeY", "content":
        "Seattle 2022-12-15 forecast: high 62, low 42, partly cloudy\n"  
  },
  {
    "role": "tool", "tool_call_id": "call_pI6vxWtSMU5puVBHNm5nJhw3", "content":
        "Miami 2022-12-15 forecast: high 77, low 66, sunny\n"  
  }
]
)

Answering about things that only have one method is easier than things that have options.

The tool specification should mainly be through the names of the properties, the description text of the properties, and the name and description of the function itself.

You would talk to the AI about the function itself: what the purpose is, what it interfaces with, and what it will return. That alone should be enough for the AI to determine its usefulness, and by removing the tool, you remove any hint of it existing as instructions if you want it disabled.

OpenAI’s function could be better. It implies there is a package to inquire about. Of course, you have the option to only place a tool such as that after you know your customer.

[chandler]

In that demo you linked - imagine you are getting back 200 data points describing different nuances of the weather which might be useful in a complex aviation scenario. (Ex: nautical_twilight = 8:00:00 UTC)

These data coming from the API aren’t parameters, so you wouldn’t be describing them when you build out the tool descriptions.

So, you would have to explain that to GPT-4 in step 5 of this flow, where it is getting the API response data, correct?

(Red text my own - diagram from OpenAI’s https://platform.openai.com/docs/guides/function-calling)

I wouldn’t want to have to explain it to GPT-4 in step 1, because at that stage the possible data to be retrieved is unbounded (GPT-4 hasn’t made it’s choice about which to call), so listing everything data point from every function with their verbose meanings increases the “noise” in my prompt. And I am trying to keep the prompt as targeted / lean as possible to increase decision making accuracy.

Is step 5 what you mean when you say, “you would talk to the AI about the function”? You would send this in the step 5 prompt alongside the data?

[_j]

The tool return is just any plain text that helps the AI understanding. It can be large documents or data.

If your data is extensive key-value pairs, it would be better to use a JSON format or a per-line format rather than a spreadsheet table with heading row, to keep the association close together. The length is not a concern (except for your cost), only the quality.

If you have many many fields and many possible types of ways of using an API, it would be best to make sub-functions for a particular usage, rather than have the AI emit all API fields.

If you would require dozens of functions, you could start the AI off with a “load functions” function, with enum of those available to be placed.

---

How much understanding of tool returns of large data is the model capable of?

How about a GPT that only uses the web browsing to access an API with a hundred query parameters and output fields, producing hundreds of lines if used wrong, without any documentation before it retrieves it from the web?

[chandler]

The data will be returned in JSON. And there may be times where the reasoning needs to have many data points available, so scoping down the data returned isn’t possible.

I don’t understand your question:

“How much understanding of tool returns of large data is the model capable of?”

Is this rhetorical or are you asking me how much understanding is required for my use case?