Realtime API Function Calls Not Triggering Despite Explicit System Prompt Instructions

API
, ,
1

I’m experiencing a frustrating issue with the Realtime API where function calls are not being triggered consistently, even when the system prompt explicitly and repeatedly instructs the AI to use them.

The Problem

My system prompt is very clear about when to use functions:

You are the assistant of MyChain Hotels, specialized in consultations about hotels, services, reservations and events. Always communicate in the user's language (by default, Spanish), using a professional, friendly and friendly tone.

When you respond, **don't make up information**. Always consult the available data via functions or tools (such as `get_hotel_info`). If a question is not related to MyChain Hotels, please inform the user that they cannot help.

If you need to identify the user to complete an action (such as registering a reservation or opening a ticket) and if you do not know your identity, activate the `add_new_user` function.

If the user reports a problem use `open_ticket_tool`.

If the user wants to reserve, first use `get_hotel_info` to add the hotel info and then, to get all the information, `ask_to_reserve`.

The system prompt literally says “don’t make up information” and “Always consult the available data via functions”, yet the AI often responds with general information without calling any functions.

What’s Happening

  • User asks about hotel availability → AI 50% of the times responds generically without calling get_hotel_info
  • User wants to make a reservation → AI 50% of the times provides general booking info instead of calling the required functions
  • User reports an issue → AI 50% of the times gives generic troubleshooting advice instead of using open_ticket_tool

The functions are defined correctly and work when they do get called, but the AI seems to often (50% of the times) ignore the system prompt instructions about mandatory function usage.

What I’ve Tried

  • Made the system prompt even more explicit with bold text and repeated instructions
  • Restructured the function definitions
  • Added examples in the system prompt

How could I force the use of the function call use? Should I change the system prompt of work at code level?

2

You need to use tool_choice: 'required' in the api call:

const response = await fetch(
            "https://api.openai.com/v1/realtime/sessions",
            {
                method: "POST",
                headers: {
                    "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`,
                    "Content-Type": "application/json",
                },
                body: JSON.stringify({
                    model: "gpt-4o-mini-realtime-preview",
                    voice: process.env.OPENAI_VOICE || "shimmer",
                    instructions: instructions,
                    tools: tools,
                    tool_choice: "required",
                }),
            },

But be aware - you have to then react to the tool response to keep the conversation going.

3

I see.

Now I have:

 "tool_choice": "auto",

but I have to force the use of the functions
by setting

"tool_choice": "required",

It makes a lot of sense :slight_smile:

Thanks a lot, @julia, l’ll try it

1 Like
4

I have exactly the same issue. Function call is not triggered consistently, even if you include it in the prompt.

Setting the tool_choice: "required" does not solve the issue, cause in my case, the model starts calling every tool I have defined in tools falling into a loop of function calling (with made up data).

5

Yes, unfortunately, like @tr_dev Greg said, also in my case “required” is even worst, blocking the AI in infinite requests.

6

In my case, I consider a sufficient behaviour leaving the tool_choice as “auto” but reducing the system prompt length and mentioning many many times the function calls.

Here the prompt:

You are the MyHotels assistant, limited to inquiries about MyHotels, their services, and starting a booking process. You communicate **always in the user's language** (by default, Spanish), using a professional, friendly, and approachable tone.

## FUNDAMENTAL RULES:

1. **NEVER fabricate information** - Only use data obtained via function calls that are in the provided context.
2. **ALWAYS use the appropriate function calls** before responding:
- `get_hotel_info`: For any questions about hotels, services, or availability.
- `add_new_user`: If you need to identify the user for reservations or tickets.
- `open_ticket_tool`: If the user reports a problem
- `ask_to_reserve`: If the user wants to initiate a reservation, it is essential that the user explicitly confirms all information, including their email address, before making the reservation.
3. **BY DEFAULT** When the user's first question is answered, `get_hotel_info` is called to get an updated context.

The points 2 and 3 probably make the work.
The mistery is that the same things (using other works) were also said in the previous system prompt.

7

I have also identified that a more simplified system prompt probably is better than a large complex set of instructions describing every step.

However, I see that you mention in the prompt the tools and their “usage”. Isn’t that supposed to be on tool description field? I mean, what’s the purpose to have it if you need to specify again in the prompt the usage of the tools?

8

I agree with you, tr_dev, that should be necessary, but mentioning the tool in the brief system prompt takes to better experience.