GPT3.5 Function Calling - Handle out-of-context queries

API
, , ,
1

I am currently developing a chatbot capable of executing various functions based on user queries.

Let’s say the chatbot currently supports two functions - book_appointment and fetch_records. The chatbot effectively handles user queries related to these functions.

However, it currently lacks context awareness when handling queries unrelated to these specific functions. For example, if a user asks about applying for a job, connecting to an agent, or checking the order status, the chatbot is responding inappropriately.

Request

{
    "model": "gpt-3.5-turbo-0613",
    "messages": [
        {
            "role": "system",
            "content": "Today is Tuesday, 01th August 2023 11:09 AM."
        },
        {
            "role": "user",
            "content": "I want to apply for a job"
        }
    ],
    "functions": [
        {
            "name": "book_appointment",
            "description": "Use this function to book an appointment",
            "parameters": {
                "type": "object",
                "properties": {
                    "user_id": {
                        "type": "string",
                        "description": "user id"
                    },
                    "date": {
                        "type": "string",
                        "description": "date of appointment"
                    }
                },
                "required": [
                    "user_id",
                    "date"
                ]
            }
        },
        {
            "name": "fetch_records",
            "description": "Use this function to fetch records",
            "parameters": {
                "type": "object",
                "properties": {
                    "user_id": {
                        "type": "string",
                        "description": "user-id"
                    },
                    "start_date": {
                        "type": "string",
                        "description": "start date of records"
                    },
                    "end_date": {
                        "type": "string",
                        "description": "end date of records"
                    }
                },
                "required": [
                    "user_id",
                    "start_date",
                    "end_date"
                ]
            }
        }
    ],
    "function_call": "auto",
    "temperature": 0,
    "max_tokens": 2048,
    "stream": false,
    "top_p": 1,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "stop": null
}

Response

{
    "id": "chatcmpl-7iiXRB7f65HqiK379PfgHaR4MOwON",
    "object": "chat.completion",
    "created": 1690891993,
    "model": "gpt-3.5-turbo-0613",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "That's great! I can help you with that. Could you please provide me with some more information? What kind of job are you looking for?"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 136,
        "completion_tokens": 31,
        "total_tokens": 167
    }
}

I have one solution, implementing a default function named others that the chatbot will call for out-of-context queries. Any suggestion on this?

2

How about just expanding your system prompt?

I tried your functions as is and updated the system prompt.

const system_prompt = `You are a helpful customer support chatbot for ACME company.\n` +
        `Always greet the user and be polite.\n` +
        `If user wants to book an appointment, call book_appointment function.\n` +
        `If user requests some records, call fetch_records function.\n` +
        `If the user wants to apply for job, tell user to contact the ACME Hiring department.\n` +
        `If the user wants to avail of products or services, tell user to contact the ACME Products department.\n` +
        `For all other inquiries, tell the user to call the helpline at 0917-1234-5678 or via email at helpdesk@acme.com.\n` +
        `Today is ${new Date()}`

Here are some of the results.

  1. Query: I want to apply for job
    Response:
{
  role: 'assistant',
  content: 'Thank you for your interest in working with ACME! To apply for a job, please contact our ACME Hiring department. They will provide you with all the necessary information and guide you through the application process.'
}
  1. Query: hi do you provide training services?
    Response:
{
  role: 'assistant',
  content: 'Yes, we provide training services. For more information, please contact our ACME Products department.'
}
  1. Query: hello how can i get hold of Mike Wazowski of the Sales department?
    Response:
{
  role: 'assistant',
  content: "Hello! I'm sorry, but I don't have access to specific employee contact information. However, if you would like to get in touch with the Sales department, you can contact our helpline at 0917-1234-5678 or send an email to helpdesk@acme.com. They will be able to assist you further."
}
3

In my experience, the function call description of a function tends to act “additively” in a very permissive manner. If you want restrictions on the “book appointment” function, or universe of queries the bot answers, that needs to be specified in your system message prompt.

Tell it specifically under what circumstances to call those functions (e.g. what kind of appointment?). Also include the kinds of questions the chatbot should and shouldn’t respond to. If it isn’t supposed to answer Q’s about connecting to an agent, then add a universe of question types it is supposed to answer. And explicit instructions to decline other types of inquiries.

Be aware, it is very easy to jailbreak these by giving the bot a “double-alignment” instruction, for example, saying you need an Agent’s phone number in order to book an appointment, then it will comply with the former so that you can achieve the latter.

If you have this bot script looping and it resets to a neutral state after a function call, then a dummy function to get called as a “catch” for out-of-scope behavior is a viable option to add a bit of robustness to the behavior. e.g. if it takes more than a few exchanges to break the bot from it’s intended behavior, the dummy function will jump in and reset the conversation.

4

I would try adding a little more detail in the function description so can be more relatable to the query. Maybe “Use this function to book an appointment in regards to applying for job or work”