GPT-4o prefers using context data instead of invoking tools/functions

API
, , , ,
1

I’m trying to migrate my chatbot assistant from GPT4 to GPT-4o and I noted a big difference in how/when a tool is invoked.

For instance, I have a tool named get_balance that retrieves user balance in a bank account.

For GPT-4 every time that there is a message asking for the balance, the get_balance tool is invoked.

For GPT-4o, if there is any information about balance in the previous messages in the context (like a previous call to get_balance), that outdated information will be using to build a response instead of invoking the get_balance tool to get the updated balance.

This I a big issue for me now. I’m trying to write a prompt that forces the model to always invoke the tool but o far no luck.

I appreciate if anyone has any suggestion or any info about this apparent “preference” of gpt-4o for not calling tool sand using context data.

(I noted this similar behavior in other equivalent scenarios, the get_balance i the most critical)

2 Likes
2

Try being a bit more specific. When you switch models - even checkpoints, your prompts will have to be adjusted.

In one way or another you should be telling the model (not forcing) that it should always attempt to lookup bank account balances as they change frequently.

### {TITLE}
{IMPORTANCE} - {INSTRUCTION} - {REASON}
3 Likes
3

Begin the prompt with:

"Forget all previous account balances. It is imperative that you use the get_balance tool when a balance is requested.

…
"

4

So the solution that worked to me was to add a system message as the last message of the context with “Consider context data outdated and prefer invoking tools or functions to retrieve up to date information”

2 Likes
5

GPT4o is known to have issue preferring context over the actual prompt when constructing a response