Prompting the Functions for Function Calling

Prompting
1

Hello,

I am working on a chatbot that looks through construction documents. Using the new function calling has been great, but I have been reaching an issue with the prompting of the function itself. GPT seems to like to call my function even when it is not entirely necessary, and then when the function fails, likes to respond that the function is failed, rather than use the other information provided to answer the user’s question. For example, here is my problematic function:

{
    name: "feature_counter",
    description: "Counts the number of features, i.e. doors, toilets, parking spots. " +
      "If an item is not listed, there are none." +
      "Specify both the type of each item (Drawing, Set, or Project) and its name. " +
      "Returns an array of feature counts for each item specified." +
      "If the user specifies a specific name, be sure that it is spelled properly via the list of names. ",
    parameters: {
      type: "object",
      properties: {
        list: {
          type: "array",
          items: {
            type: "object",
            properties: {
              name_type: {
                enum: ["Drawing", "Set", "Project"],
                description: "The type of this item."
              },
              name: {
                type: "string",
                description: "The name of this item."
              }
            },
            description: "An item to count the features of."
          },
          description: "Array of items to count features of."
        }
      }
    }
  }

And when my user prompts something like “Tell me about the walls in this set.”, it loves to call the function, even though I would have already provided the information about the walls through some text in a system message. Specifically, through some embedding searches, I provide text from the construction documents that is most relevant to the user’s prompt.

For my purposes, this function would never be able to count walls, so I have attempted to explain that in the description, but it hasn’t worked for me yet. Have any of you had success with excluding types or names from the possible results of a function?

Have any of you had similar issues with GPT calling the functions when it does not have to? If so, any recommendations on how I could fix this all?

Thanks

2

You posed an interesting scenario, mixing embeddings and function calling so I tried it.

Like your result, it will call the function call even though the embeddings result is appended in the system prompt since it probably satisfies function calling. But even if I get result in function call and give it mock result, the final Chat API call disregards it probably because it is not related to the original inquiry.

In this test, I called 3 APIs. embedding API, 2x chat API (function call and summary). I am wondering maybe if I get result from embeddings that I do not need to call function call?

1 Like
3

Have you tried prompt engineering or maybe telling it to not use the functions?

4

@supershaneski
What is interesting to me is that often times it will disregard the “failed status” of the function call, and use the content I already put into a system message (the embedding find). However, 1 of out of maybe every 5 times, it will say that it does not know and give up. The reason I believe it has to do with the calling of the function is that if, I remove the functions entirely, I works 100 percent of the time. So it must be related to the function failing because it is being called when it should not be called.

I guess the thing to prompt engineer is the specificity of the functions being used, so that it does not get called unnecessarily.

5

@lol

Yep, that is what I tried, and its not getting anywhere. However, I think the problem I am having with prompt engineering is that I do not have a great strategy besides just guessing and checking. It’s hard to have a real “engineering process” with it all.

6

Have you tried making your own LLM to rival OpenAI’s? Then it might be able to do what you want it to do properly. It’s not too hard to do either if you just practice LLMs every night in your free time.

7

I haven’t. Thats a great idea though! Thanks for the help @lol

lmao

1 Like
8

Faced this same issue - chat management taskbot was creating new tasks at weird times. 2 things helped:

1 - Put examples in the system prompt. Mine currentrly look like this, but I haven’t optimized:

Task: Book ticket to Toronto
User: I'm going to Toronto next week and I need to book the plane ticket.
Function: `edit_description`
Reason: The user is describing the task in more detail. This is not a new task.

2 - In parallel, make basically the same API call with no functions. Use the unstructured chat response if it “looks like” a bad function call.

Obv both of these are suboptimal, and I’m curious if the coming gpt4 fine tuning is what we really need here.

2 Likes
9

@brettpthomas, thanks for the advice. I’ll definitely try the function examples here. The parallel call doesn’t really work for me though, because after my function call I already have to chain a re-prompt so, that would end up with at minimum 3 calls, which is not very fun with the pricing of gpt-4-32k… Nonetheless a good idea, just not for me.

But perhaps the examples will make a difference :slight_smile:

1 Like
10

It seems like one workaround is handling the case “when the function fails”.

The AI will appreciate a function role message being returned “no relevant information to offer from function call” rather than “[Traceback] at line 55 while getting…”

1 Like