Suggestion: function-calling using limited information for saving tokens

API
, , , ,
1

I am in love with the function-calling feature introduced in gpt-3.5-turbo-0613, and it’s been a game-changer for my GPT-3.5-Turbo bot. I have a lot of functions, though, and the problem is the token usage. Even if you only include one function, it still costs a lot of tokens – say, 400 to 550 tokens per function in my testing. High token usage, especially with multiple functions, can cost you a lot more money, and it’ll also reduce the number of tokens available for responses in total.

Because of this, I’ve had to implement conditional checks to see which functions are added to the functions that my bot is able to call. For instance, if certain words or phrases exist in a user’s message, a relevant function will be available for my bot.

The issue with this is that my bot won’t always be able to call the correct function.

A feature for function-calling that would be fantastic is a way to simply pass the function names to the bot – perhaps with short descriptions. Say, GPT-3.5 only sees image_generation as a function and is able to call it, and if it decides to call it, it is then able to fetch its description, parameter information, etc. to call it with arguments.

As far as I’m aware, this is not possible currently. If there is a way to do this, please let me know! I suppose it would be possible by sending a regular response first to get the correct function before generating a function-call response, but that wouldn’t be ideal since it’d require an additional response before the function-call and wouldn’t be perfect.

1 Like
2

You could create a function that loads other specific function(s) and its description on the API call based on certain criteria, and let the model decide what function it wants to use.

This shouldn’t use much tokens as the function description will on contain 2 params, i.e the list of available functions and the part of the user message to call the function on.

This 2-step function is only economical if you have more than 2 functions that you want the model to be able to choose from.

4 Likes
3

Thank you. Can you elaborate on what you mean by this? Firstly, what do you mean by ‘the part of the user message to call the function on’?

Also, I do have a function for selecting functions based on certain criteria, such as phrases or words found in the user message. However, this does not let the model decide what function to call between all available functions, so I’m assuming this isn’t what you mean.

After thinking about it for a little bit, I came up with this, which is an example of what I believe you might be referring to:

'get_function':
    {
        'name': 'get_function',
        'description': 'Get relevant function to call, if any',
        'parameters': {
            'type': 'object',
            'properties': {
                'name': {
                    'type': 'string',
                    'description': 'Name of function to call. List: generate_image, generate_text, generate_video, text_to_speech, translate, get_timezone.',
                },
                'info': {
                    'type': 'string',
                    'description': 'Information from user message to pass to chosen function.'
                }
            },
            'required': ['name'],
        }
    }

The current issue I have is that I can either choose to pass many functions to the GPT-3.5 model, costing a lot of tokens, or conditionally selecting which functions to pass but losing the ability for the model to dynamically pick which function to call.

If I simply passed the above function to the model and no other functions, it would let the model decide between any of the functions, assuming that get_function() is set up to be able to run any of the function described in the description string of the name property.

The problem, though, is that it doesn’t get the description for the function it wants to call. Let’s say I have a function to generate an concise image based on a given query; I’d have to describe this in the description, which would cost tokens and would only matter should the model wish to call this function. If I were to add a description for each function here, it would result in a similar problem of high token usage.

I suppose I could conditionally add descriptions instead of conditionally passing functions to the model, and it would probably save some tokens, but what I was suggesting was some sort of built-in solution, where the model is simply fed the function names – perhaps with brief descriptions – and once it decides to use a model, it then gets the full model description and parameters.

If this isn’t what you were referring to, please let me know!

1 Like
4

Here is a sample way to handle this.

Suppose we have 3 functions

{
        name: 'get_current_weather', 
        description: 'Get the current weather in a given location', 
        parameters: {
            type: 'object', 
            properties: {
                location: {
                    type: 'string', 
                    description: 'The city and state, e.g. San Francisco, CA'
                },
                unit: {
                    type: 'string', 
                    enum: ['celsius', 'fahrenheit']
                }
            },
            required: ['location',]
        }
    },
    {
        name: 'send_email',
        description: 'Write and send email',
        parameters: {
            type: 'object',
            properties: {
                to: {
                    type: 'string',
                    description: 'The recipient of the email, e.g. John, beth@email.com'
                },
                body: {
                    type: 'string',
                    description: 'The body of the email'
                }
            },
            required: ['to', 'body']
        }
    },
    {
        name: 'get_person_data',
        description: 'Get personal data of the given person',
        parameters: {
            type: 'object',
            properties: {
                name: {
                    type: 'string',
                    description: 'Name of the person, e.g. John, Florence'
                }
            },
            required: ['name']
        }
    }

Since we do not want to include all the function definitions in the API call, we create another function, the one function to rule them all call which one is actually required.

{
        name: 'get_user_inquiry',
        description: 'Get the type of inquiry based from the user message',
        parameters: {
            type: 'object',
            properties: {
                inquiry: {
                    type: 'string',
                    enum: ['weather', 'send_email', 'get_person_data']
                },
                params: {
                    type: "array",
                    items: {
                        type: "string",
                    }
                }
            },
            required: ['inquiry']
        }
    }

Here is the result of sample message and response

Inquiry: what is the weather in Tokyo today?

Response:

{
  role: 'assistant',
  content: null,
  function_call: {
    name: 'get_user_inquiry',
    arguments: '{\n  "inquiry": "weather",\n  "params": ["Tokyo"]\n}'
  }
}

Inquiry: tell me the birthday of Julia

Response:

{
  role: 'assistant',
  content: null,
  function_call: {
    name: 'get_user_inquiry',
    arguments: '{\n  "inquiry": "get_person_data",\n  "params": ["Julia"]\n}'
  }
}

Inquiry: send email to Ricky and remind him of our meeting this afternoon at 3pm in the cafeteria

Response:

{
  role: 'assistant',
  content: null,
  function_call: {
    name: 'get_user_inquiry',
    arguments: '{\n' +
      '  "inquiry": "send_email",\n' +
      '  "params": ["Ricky", "meeting reminder", "this afternoon at 3pm in the cafeteria"]\n' +
      '}'
  }
}
4 Likes
5

Thanks for your help.

I am a bit confused, however. So, in my test Python script, I pass get_user_inquiry to the model. That’s the only function it can choose between. It’s able to choose between the strings in enum, each of which represents a function.

I’m confused, though, about how I would get the model to call the function with the correct arguments, as what the arguments can be and how many there are, as well as which are required and what the defaults are, differ from function to function. The model also wouldn’t have the description of each function to tell it when to call it and what the parameters represent and expect, which is one of the issues I detailed – one whose primary solution involves using a lot of tokens. If you could provide an example, that would be much appreciated!

6

When I was making the generic function I was thinking

  • will this function prepare all the parameters required to run the selected function?

  • or, will this just give you the idea which function will be called based from user inquiry and you should run the function call again using only the selected function?

Anyhow, I think (although I have not tested) you can describe all the parameters for your different functions in the “params” instead of just generic “array” string, something like this:

...
params: {
          type: "array",
          items: {
              type: "object",
              properties: {
                  location: { type: "string" },
                  name: { type: "string" },
                  to: { type: "string" },
                  body: { type: "string" },
              }
          }
      }
  }
7

Do you mean something like this?

functions = [
{
    'name': 'get_user_inquiry',
    'description': 'Get the type of inquiry based from the user message',
    'parameters': {
        'type': 'object',
        'properties': {
            'inquiry': {
                'type': 'string',
                'enum': ['get_current_weather', 'get_birthday', 'get_dall_e_prompt']
            },
            'params': {
                'type': 'array',
                'items': {
                    'type': 'object',
                    'properties': {
                        'get_current_weather': { 'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA, and the unit (Celsius or Fahrenheit)' },
                        'get_birthday': { 'type': 'string', 'description': 'The name of the person' },
                        'get_dall_e_prompt': { 'type': 'string', 'description': 'The kind of prompt (text, image, or text and image) and the image size (256, 512, or 1024)' },
                    }
                }
            }
        },
        'required': ['inquiry']
    }}]

Thank you again for the help. Please let me know if I’m going about this incorrectly. Should I format it differently if the function expects multiple parameters? Please let me know!

8

I tested your code and it works. I would have named the parameters differently but it seems the AI understands it. So I think it is okay.

As for multiple parameters, you need to make separate entry. For example, we update the get_current_weather to get_weather and require date parameter, so we now have

{
        name: 'get_user_inquiry',
        description: 'Get the type of inquiry based from the user message',
        parameters: {
            type: 'object',
            properties: {
                inquiry: {
                    type: 'string',
                    enum: ['get_weather', 'get_birthday', 'get_dall_e_prompt']
                },
                params: {
                    type: "array",
                    items: {
                        type: "object",
                        properties: {
                            get_weather_location: { type: 'string', description: 'The city and state, e.g. San Francisco, CA' },
                            get_weather_date: { type: 'string', description: 'The date, e.g. today, tomorrow, Tuesday, 2023-07-26' },
                            get_birthday: { type: 'string', description: 'The name of the person' },
                            get_dall_e_prompt: { type: 'string', description: 'The kind of prompt (text, image, or text and image) and the image size (256, 512, or 1024)' },
                        }
                    }
                }
            },
            required: ['inquiry']
        }
 }

I separated the weather parameters, get_weather_date for the date and get_weather_location for the location .

Testing it with this inquiry:
What is the weather in Tokyo on Friday?

we get this response

{
    name: 'get_user_inquiry',
    arguments: '{\n' +
      '  "inquiry": "get_weather",\n' +
      '  "params": [\n' +
      '    {\n' +
      '      "get_weather_location": "Tokyo",\n' +
      '      "get_weather_date": "Friday"\n' +
      '    }\n' +
      '  ]\n' +
      '}'
  }
}
9

When I tested this, the model wouldn’t always call the correct functions, and it wouldn’t always call a function. For instance, if the user asks a question the model doesn’t know the answer to, it would often say it doesn’t know the answer to it, instead of calling the search function.

functions = [
    {
        'name': 'get_user_inquiry',
        'description': "Call function based on user msg. Functions:\ngenerate_image: make img with DALL-E\nmeme: gen meme based on user msg\nimages: find img on web\nget_time: get current time in location\nquiet: temp-mute user\nnews: get latest news on topic\nscrape: scrape url\nsearch: search Google (use if you don't know answer to question)\nmemory: add info to your memory\nimage_recog: analyze img for labels and text\nai_help: get info for commands/settings\nlyrics: get lyrics for song\nspotify: search spotify, get recommendations, or get info about song/artist/album/playlist",
        'parameters': {
            'type': 'object',
            'properties': {
                'inquiry': {
                    'type': 'string',
                    'enum': list(functions_map.keys())
                },
                'params': {
                    'type': "array",
                    'items': {
                        'type': "object",
                        'properties': {
                            'prompt': { 'type': 'string', 'description': 'Prompt for the img gen-' },
                            'size': { 'type': 'string', 'description': 'Size for img gen' },
                            'image_query': { 'type': 'string', 'description': 'Meme query for img' },
                            'query': { 'type': 'string', 'description': 'Query for image or news search' },
                            'first': { 'type': 'boolean', 'description': 'Return first img result, or limit Spotify result to first'},
                            'location': { 'type': 'string', 'description': 'Location for getting time' },
                            'user': { 'type': 'string', 'description': 'User to quiet' },
                            'expires': { 'type': 'integer', 'description': 'Seconds to quiet' },
                            'num': { 'type': 'integer', 'description': 'Num of news articles or search results to return' },
                            'url': { 'type': 'string', 'description': 'URL to scrape' },
                            'next': { 'type': 'boolean', 'description': 'Whether to get both num/first search results or both num/first and following result' },
                            'info': { 'type': 'string', 'description': 'Info to store in memory' },
                            'permanent': { 'type': 'boolean', 'description': 'Whether info stored permanently' },
                            'r': { 'type': 'string', 'description': 'True to analyze img' },
                            'input_cmds': { 'type': 'string', 'description': 'Input commands to fetch info for' },
                            'input_sets': { 'type': 'string', 'description': 'Input settings to fetch info for' },
                            'song': { 'type': 'string', 'description': 'Song to fetch lyrics for' },
                            'line': { 'type': 'string', 'description': 'Line to search for lyrics around in song lyrics' },
                            'kind': { 'type': 'string', 'description': 'What to search for on Spotify (song, artist, playlist, album)' },
                            'recommendations': { 'type': 'boolean', 'description': 'Whether to get recommendations on Spotify' },
                            'get_info': { 'type': 'boolean', 'description': 'Whether to get information on given Spotify song/artist/playlist/album' }
                        }
                    }
                }
            },
            'required': ['inquiry']
        }
    }
]

I’ve managed to finish the implementation of this, but I can’t get it to work as I was hoping it could work. The main problem I think is the lack of descriptions for each function, but I tried to remedy this by describing each function in the description string of this function.

I suppose the model might be confused because it sees one single function with a lot of different parameters?

10

The model is confused because it doesn’t receive the description for properties within a subarray. You’d have to name them something like image_generator_overlay_text, image_generator_size_pixels, image_generator_meme_template_query, assigned to better-named operations.

Your function, transformed into the text the AI receives, is unprecedented chaos (and I even added just some enums and description of enums myself)

(system message)

# Tools

## functions

namespace functions {

// Call function based on user msg. Functions:
// generate_image: make img with DALL-E
// meme: gen meme based on user msg
// images: find img on web
// get_time: get current time in location
// quiet: temp-mute user
// news: get latest news on topic
// scrape: scrape url
// search: search Google (use if you don't know answer to question)
// memory: add info to your memory
// image_recog: analyze img for labels and text
// ai_help: get info for commands/settings
// lyrics: get lyrics for song
// spotify: search spotify, get recommendations, or get info about song/artist/album/playlist
type get_user_inquiry = (_: {
// select function to invoke (added_j)
inquiry?: "generate_image" | "meme" | "images" | "get_time" | "quiet" | "nnews" | "created_truncated_j",
params?: {
  prompt: string,
  size: string,
  image_query: string,
  query: string,
  first: boolean,
  location: string,
  user: string,
  expires: number,
  num: number,
  url: string,
  next: boolean,
  info: string,
  permanent: boolean,
  r: string,
  input_cmds: string,
  input_sets: string,
  song: string,
  line: string,
  kind: string,
  recommendations: boolean,
  get_info: boolean,
}[],
}) => any;

} // namespace functions

Also calling them “functions”, within a function, within the function block is definitely formula for confusion. “specify operation perfomed based on…” and replace “inquiry” with “operation_selected”

I would make a one-line description in simulated JSON:
"All operations available: {generate_image: AI-created from description, generate_meme: from popular template plus user text, … }

Many of the abbreviations you came up with save no tokens.

move all those function parameters out of the subarray to where they can be described, and give them names like:
operation_get_time_null
operation_lyric_finder_known_phrase

And then, after reaching the pinnacle of optimization, becoming now about the same size as individual functions, I don’t think it’s going to work well.

11

have you explored the guidance library yet?

12

Thanks. I suppose it’s a lost cause, which is why I’d love a feature for this implemented in gpt-3.5-turbo: feed the model names and perhaps brief descriptions of the functions without requiring parameters – then once it decides a function, give it the full descriptions and parameter descriptions.

What shortened forms in particular do not save tokens?

13

Like msg, img, gen. You can use the full word and then have better understanding also.

image

14

It looks like you’re describing multiple functions (get_time(), lyrics(), etc) so the AI can choose one based on the user prompt. But you’re cramming them all into one giant function rather than submitting them as a collection of individual functions for the AI to browse and select from.

I guess it depends what your limitations are. If it absolutely HAS TO be done in one call, then you have to spend the tokens to provide the full descriptions in the context. If the priority is low token spending, bite the bullet and do two calls, one with lightweight descriptions and let ai pick one, THEN do another call with the full-fat description for just its choice.

15

The way I do function-calling is to use functions to select what functions to send to the model for function-calling. These function detect key words or phrases, such as ‘Spotify’, ‘search’, ‘lyrics’, and so on. This is not ideal because users may use different words or write in a different language. However, feeding each function to the model costs a lot of tokens, costing more money and resulting in longer response times.

I haven’t experimented with doing two calls yet. If anyone has a suggestion on how to achieve this as efficiently as possible, please let me know. Concerned with token usage, response times, and especially rate limits.

16

Edit: with multiple function calling and assistant tools nwo this is probably obsolate

I think embeddings are calculaetd sparately in terms of rate limits, so if you’re especially concerned about rate limits you could make your first step to get an embedding of the users query and compare it locally to embeddings of your features (play music, generate image, whatever). And use that in place of a GPT4 call to select which function. Then only hit GPT4 once, for it to fill out the function call for the feature selected via embeddings values.

eg:
done once and saved for later:
“spotify” → embeddings service → [0.015, …]
“lyrics” → embeddings service → [0.521, …]
“search” → embeddings service → [0.332, …]

done every user request:
“im feeling moody” → embeddings service → [0.3879, …]
[0.3879, …] x [0.015…] = 0.76
[0.3879, …] x [0.521…] = 0.22
[0.3879, …] x [0.332…] = 0.4
0.76 (comparison of user query against “spotify”) is the higest score

then fill out your gpt4 request with your spotify function definition, add the user prompt, and hopefully gpt will return back the string you can use to call spotify and get a moody result.

(or use a vector embedding service)

As for making multiple calls, it depends on how you’re making the calls to OpenAI. Like do you have an actual application running in a server somewhere? Or something more like webhooks calling cloud functions.

1 Like
17

Try to extend the way you are doing right now by using array in the keyword detection property so that it can return all the possible keywords.

18

You can use this blog for help.
It lays out techniques to overcome the token limitation for function calls.

2 Likes
19

@pratham How is this approach working for you?

20

It’s pretty good.
Embedding models are as good as GPT in finding the most relevant function based on user queries.
So you will always have just 2-3 most relevant functions in your LLM call, saving you a lot of token size without any visible loss to the quality of results. (In my case, I never saw a single incorrectness from Embedding model but quite a few from GPT)

I have tested it with about 15-20 functions, and it works very well. But I believe, theoretically, you can have as many functions as you want, given they are not too similar but sufficiently distinct and have proper descriptions.