Chat completion api tool call loops

API
,
1

General overview

I have a c# .net 8 Application where I am trying to implement a chat like behaviour using the OpenAI API with GPT4o Model

I have a collection (chatHistory) where I store every message be it of role ‘system’, ‘assistant’, ‘function’ or ‘user’

Also I have currently 2 functions:

  1. CreateLiquidityPlan
    - Generates a liquidity plan as an excel file for the specified years
  2. CurrentLiquidityStatus
    - Provides information on the liquidity status on a specific date with regard to a specific event

CreateLiquidityPlan should be called on requests like “generate the liquidity plans for 2024 and 2025”

CurrentLiquidityStatus should be called on requests like “Where will we be at the end of the year when the order of Mr.X comes in?”

Execution

When I create a new conversation a new chatHistory is created and a System Prompt with general instructions like “Answer in the same language as the user asked.” is appended.

Then the user can do his prompt → “Where will we be at the end of the year when the order of Mr.X comes in?”

role content
system Answer in the same language as the user asked.
user Where will we be at the end of the year when the order of Mr.X comes in?

The chat history is then sent to the ChatCompletion API which returns the request of a tool_call.

So i figure out which tool I should call, execute the code and I append a Function!!! message to the history containing the result.

role content
system Answer in the same language as the user asked.
user Where will we be at the end of the year when the order of Mr.X comes in?
function [CurrentLiquidityStatus] - If the following occurs: ‘Mr.X order’
Then the following happens: ‘We end up with an annual surplus of 100.295,53 euros on 2024-12-31’.
TODO: Generate a liquidity plan as an excel file for the specified year 2024

The history is again sent to the api…

Expectation

I now expect that because of the “TODO” the CreateLiquidityPlan should be requested so i can append the result to the chat history and after calling the ChatCompletion API again it will stop requesting tool_calls and finally create a assistant message.

role content
system Answer in the same language as the user asked.
user Where will we be at the end of the year when the order of Mr.X comes in?
function [CurrentLiquidityStatus] - If the following occurs: ‘Mr.X order’
Then the following happens: ‘We end up with an annual surplus of 100.295,53 euros on 2024-12-31’.
TODO: Generate a liquidity plan as an excel file for the specified year 2024
function [CreateLiquidityPlan] - A liquidity plan for the year 2024 was successfully created!
assistant When the order of Mr.X comes in, we end up with an annual surplus of 100.295,53 euros on 2024-12-31. The liquidity plan for the year 2024 was created!

Reality

OpenAI requests to call CurrentLiquidityStatus over and over again… So I built in a
loop detection but I think I’ve made a mistake somewhere in my thinking about how the whole function calling thing is supposed to work.
I would be grateful if someone here could help me as I have been struggling with this problem for some time now

2

:thinking:

Do I get this right?

  1. The model calls CreateLiquidityPlan
  2. the response from CreateLiquidityPlan is
    • “a, b, c, x, y, z, now create a liquidity plan”
  3. the model follows the instruction and calls CreateLiquidityPlan
  4. the response from CreateLiquidityPlan is
    • “a, b, c, x, y, z, now create a liquidity plan”
  5. the model follows the instruction and calls CreateLiquidityPlan

It seems like the model is just following instructions, unless I’m misunderstanding something :thinking:

1 Like
3

you need to loop and wait until the api finishes. however, it may happen that the api calls your function over and over again because you are not providing what it expects. like you send an inquiry: “what’s the weather in paris?” and your function returns the weather in london. also in the event something went wrong processing the tool, you also need to provide a good error message. otherwise, it will also continually just call it again and again. as a last resort, you also need some sort of max_loop check.

2 Likes
4

Sorry I wrote that wrong.

CurrentLiquidityStatus is called correctly.
CurrentLiquidityStatus returns an answer AND the TODO → “Create a Liqidity Plan”

Now I expect CreateLiquidityPlan to be called but instead CurrentLiquidityStatus is looped over and over again

1 Like
5

On top of what @supershaneski suggested, It might also be a good idea to make it more obvious what the difference between the two functions is. Either in name or description, but ideally both.

Me personally, I’d steer clear of functions altogether and just use JSON. :laughing:

1 Like
6

The description are like this:

CreateLiquidityPlan
- Generates a liquidity plan as an excel file for the specified years
CurrentLiquidityStatus
- Provides information on the liquidity status on a specific date with regard to a specific event

I dont know but for me this seems to be clear…

I have this problem with nearly every function.
E.g
I have a function GetMails that gives Mails for a specific topic.
When i prompt give me mails for the topic xyz then GetMails is called and delivers 3 mails. After that GetMails is called again and delivers the same 3 mails again and so on and so on…

It is like he is ignoring the function results that I send him

7

Here’s what I believe the issue to be:

Your CreateLiquidityPlan almost BEGS to have CurrentLiquidityStatus to be called again.

Think of it like this:

You have 2 buttons. One is “Create Plan”, the other is “Get Information”. You have a sheet of very limited information in front of you.

You want to create a plan, but have insufficient/inadequate information, so you press “Get Information”.

The information returned is unsuitable. Hmmm, maybe again?

No? Well. This isn’t sufficient. I NEED more information to create whatever this plan is. And just like a disgruntled worker, you just waste time bashing the button expecting something different to happen :rofl:

You should try being more specific in WHAT this liquidity plan is, and the amount of information required to create one. I would also think about using a separate assistant that has all the information attached via embeddings.

tl;dr: The model is probably thinking it needs more information and is not satisfied with your status responses.

1 Like
8

Oh wait, oh wait.

Are you sending the run again, where the last response in the thread is an assistant message?

There is a longstanding “bug” or weird behavior where the chat models absolutely ignore the last “assistant” message, and just repeats it. It’s possible that this is what you’re seeing.

Yeah, assistant functions are a bit of a pain to debug. I’d try clarifying the function, sending a user message in between, or just skipping the whole assistants thing altogether if possible.

1 Like
9

This sounds plausible.

In any case where my assistants decide to re-run a function call they always first post a message (“Oh, this didn’t work, let me try again”, or “The server reported that the information was insufficient, can you add more details?”) and THEN post the function call afterwards.

I completely skipped this part. It definitely feels like a user error if the model is always entering loops regardless of the context. I think at this point some code would be handy to see.

This is some serious double dipping :rofl:

10

It is possible that this is hallucination.
It is also possible that the API do not think the 3 mails submitted back covers the topic xyz.
Or, it does not know when to stop getting mails, in which case, you can design the output to specifically say there are only 3 mails found for the topic xyz.

11

You are talking about “runs”
Just to clear that out: I am NOT using assistants I use ChatCompletion so I need to send the whole chat history with every request.

12

As my C# code is a bit too customized and would lead to confusions I will try to break it down so some simple python code:

As python is not my strength excuse possible mistakes.

# GetMails Function
functions = [
    {"""
{
    "name": "GetMails",
    "description": "The function searches for user mails. Therefore the function must know if the aim is a summary or just to return an application program attachment that the user can open and view separately. The required parameters are 'ContentFilter' and 'ForSummary', all the other parameters are optional. The parameter 'ForSummary' is required and should be set to true only if the user explicitly asks for a summary of mails. Otherwise, it should be false.",
    "parameters": {
        "type": "object",
        "properties": {
            "Senders": {
                "type": "array",
                "description": "Is an optional parameter that lists email addresses. If no email address is required, an empty list is used.",
                "items": {
                    "type": "string",
                    "description": "If set, then it must be a valid email address."
                }
            },
            "FromDate": {
                "type": "string",
                "description": "Is an optional parameter that defines the start date for selecting emails. If a date is set, then it must be valid formatted to 'yyyy-mm-dd'. Depending on the current date, a date in future is not allowed."
            },
            "amount": {
                "type": "integer",
                "description": "Is an optional parameter that defines how many mails should be selected. The default value if not given is 3."
            },
            "ContentFilter": {
                "type": "string",
                "description": "Is an required parameter that is used as mail filter and can't be null or empty. It defines a filter for the content of the email body or mail subject."
            },
            "ForSummary": {
                "type": "boolean",
                "description": "Is an required parameter that is only true if the user explicitly asks for a summary of mails. Otherwise its false."
            }
        },
        "required": ["ContentFilter", "ForSummary"]
    }
}
    """}
]

# create a system prompt
system_message = """
                Always follow these instructions: 
                - Do not use any default values.             
                - Answer in the same language as the user asked.
                - Try to shorten your questions to maximum 3 sentences 50 Words.
                - For date comparisons use the date of today. Today is the 02.08.2024
                - Use the following JSON format for your response: 
                {
                "Content": "Your Response Message"
                }
                """

# user prompt
user_prompt = "Show my mails to the topic 'easter'"

# Add the system message to the chat history array
chatHistory = [
    {"role": "system", "content": system_message},
    {"role": "user", "content": user_prompt}
]

# Create the chat completion
response = openai.ChatCompletion.create(
    model='gpt-4o',
    messages=chatHistory,
    temperature=0,
    functions=functions,
    function_call="auto",
    response_format="json"
)

##############################################################################################
# response result here is:
# {
#   "choices": [
#     {
#       "message": {
#         "role": "assistant",
#         "content": null,
#         "name": null,
#         "tool_call_id": null,
#         "tool_calls": [
#           {
#             "id": "call_T81SESV5Dvocf1lRyMDY9yqg",
#             "type": "function",
#             "function": {
#               "name": "GetMail",
#               "content": null,
#               "arguments": {
#                 "ContentFilter": "Easter",
#                 "ForSummary": false
#               }
#             }
#           }
#         ],
#         "function_Call": null
#       },
#       "finish_reason": "tool_calls",
#       "index": 0
#     }
#   ],
#   "id": "chatcmpl-9rkGbHeACREzbLCNq2VRAgioWsvtf",
#   "usage": {
#     "prompt_tokens": 899,
#     "completion_tokens": 21,
#     "total_tokens": 920
#   },
#   "system_fingerprint": "fp_bc2a86f5f5",
#   "created": 1722596261,
#   "model": "gpt-4o-2024-05-13",
#   "object": "chat.completion",
#   "error": null
# }
##############################################################################################

# Execute the GetMails Function and add the result to the chatHistory
function_result = "2 mails found. The mails can be displayed by clicking on them."

# Add the function Result to chathistory with FUNCTION-role
chatHistory.append(
    {"role": "function", "content": function_result}
)

# Get the next response
response = openai.ChatCompletion.create(
    model='gpt-4o',
    messages=chatHistory,
    temperature=0,
    functions=functions,
    function_call="auto",
    response_format="json"
)

##############################################################################################
# response result here is:
# {
#   "choices": [
#     {
#       "message": {
#         "role": "assistant",
#         "content": null,
#         "name": null,
#         "tool_call_id": null,
#         "tool_calls": [
#           {
#             "id": "call_T81SESV5Dvocf1lRyMDY9yqg",
#             "type": "function",
#             "function": {
#               "name": "GetMail",
#               "content": null,
#               "arguments": {
#                 "ContentFilter": "Easter",
#                 "ForSummary": false
#               }
#             }
#           }
#         ],
#         "function_Call": null
#       },
#       "finish_reason": "tool_calls",
#       "index": 0
#     }
#   ],
#   "id": "chatcmpl-9rkGbHeACREzbLCNq2VRAgioWsvtf",
#   "usage": {
#     "prompt_tokens": 899,
#     "completion_tokens": 21,
#     "total_tokens": 920
#   },
#   "system_fingerprint": "fp_bc2a86f5f5",
#   "created": 1722596261,
#   "model": "gpt-4o-2024-05-13",
#   "object": "chat.completion",
#   "error": null
# }
##############################################################################################

# So from here it just repeats...
# I also tried to do like so:

chatHistory.append(
    response.result["choices"][0],
    {"role": "function", "content": function_result}
)

##############################################################################################
# but then i just get
# {
#   "Result": null,
#   "ErrorMessage": null,
#   "StatusCode": 200,
#   "IsSuccess": true,
#   "ErrorResponse": null
# }
##############################################################################################

# And I would at least expect a formulated message by the AI
# So when i ask my question in German and my function result is in english (what actually is the case in my app)
# Then I need the AI to take the english function result and formulate a German answer
13

There are several problems with the code.

  1. you are trying to use the (deprecated) functions and tool calls interchangeably. Only use tool calls.
  2. You are messing around with the JSON mode, and that should not be an argument sent when working with tools.
  3. You are forgetting to append all messages to the chat history.
  4. The tool response is too nebulous and will confuse the model by making it think, ‘something must’ve gone wrong because this is not the kind of response I was expecting. Let me go ahead and try calling that tool again!’
  5. The schema needs to be a python dict instead of str and it needs to be formatted for tool calling.
# GetMails Function
import openai, json

client = openai.OpenAI()

tools = [
    {
        "type": "function",
        "function": {
            "name": "GetMails",
            "description": "The function searches for user mails. Therefore the function must know if the aim is a summary or just to return an application program attachment that the user can open and view separately. The required parameters are 'ContentFilter' and 'ForSummary', all the other parameters are optional. The parameter 'ForSummary' is required and should be set to true only if the user explicitly asks for a summary of mails. Otherwise, it should be false.",
            "parameters": {
                "type": "object",
                "properties": {
                    "Senders": {
                        "type": "array",
                        "description": "Is an optional parameter that lists email addresses. If no email address is required, an empty list is used.",
                        "items": {
                            "type": "string",
                            "description": "If set, then it must be a valid email address.",
                        },
                    },
                    "FromDate": {
                        "type": "string",
                        "description": "Is an optional parameter that defines the start date for selecting emails. If a date is set, then it must be valid formatted to 'yyyy-mm-dd'. Depending on the current date, a date in future is not allowed.",
                    },
                    "amount": {
                        "type": "integer",
                        "description": "Is an optional parameter that defines how many mails should be selected. The default value if not given is 3.",
                    },
                    "ContentFilter": {
                        "type": "string",
                        "description": "Is an required parameter that is used as mail filter and can't be null or empty. It defines a filter for the content of the email body or mail subject.",
                    },
                    "ForSummary": {
                        "type": "boolean",
                        "description": "Is an required parameter that is only true if the user explicitly asks for a summary of mails. Otherwise its false.",
                    },
                },
                "required": ["ContentFilter", "ForSummary"],
            },
        },
    }
]

# create a system prompt
system_message = """
Always follow these instructions: 
- Do not use any default values.             
- Answer in the same language as the user asked.
- Try to shorten your questions to maximum 3 sentences 50 Words.
- For date comparisons use the date of today. Today is the 02.08.2024
"""

# user prompt
user_prompt = "Show my mails to the topic 'easter'"

# Add the system message to the chat history array
chat_history = [
    {"role": "system", "content": system_message},
    {"role": "user", "content": user_prompt},
]

# Don't use functions. Use tool calls instead
response = client.chat.completions.create(
    model="gpt-4o",
    messages=chat_history,
    tools=tools,
    tool_choice="required",  # for testing purposes, make sure this is set to required to force tool calls
    # functions=functions,
    # function_call="auto",
    # response_format="json",
)

# you forgot to append the tool call to the chat history, and it must be the message not the choice
message = response.choices[0].message
chat_history.append(message)


# This does not satisfy the requirements of the task. The user asked for a list of mails with the topic 'easter'.
tool_response = json.dumps(
    dict(
        success=True,
        mails=[
            "Thanks for the easter basket!",
            "The easter bunny is coming!",
            "Happy easter!",
        ],
    )
)
# Add the function Result to chathistory with FUNCTION-role
chat_history.append(
    {"role": "tool", "content": tool_response, "tool_call_id": message.tool_calls[0].id}
)

# Get the next response
response = client.chat.completions.create(
    model="gpt-4o",
    messages=chat_history,
    # temperature=0,
    # functions=functions, # you don't need to pass functions when generating user responses; save the tokens!
    # function_call="auto",
    # response_format="json", # you definitely don't want JSON format when reply to the user
)
print(response.choices[0].message.content)
2 Likes
14

I think you gave the deciding hint!
My library that i use to make the calls (Forge.OpenAI) does not offer the chatMessage role tool, only function

Since I changed my chatMessage role to tool everythiing workd like a charm.

BTW: The python code only corresponds to the basic structure of my code and should actually only serve to understand how my code proceeds sequentially to process the requests

I would still like to talk about the other points…

  1. check!
  2. What do you mean by “messing around” I feel like i need to use the ResponseFormat JSON because its hard for me to access the response otherwise. So I tell him “use JSON” and in the system prompt “use this format” so i can parse his answer
  3. When a tool choice is requested i append the “assistant” message with empty content to the history and when the function result is there I append a {role: "tool", content: function_result} to the history
  4. no more problems since 1 is fixed
  5. I dont use python :stuck_out_tongue:
3 Likes
15

Understood. The JSON mode isn’t always reliable for returning the object in the exact format you request. Its utility is limited, though it can be useful for processing an undefined structured format. However, if you require a specific format for your object, it’s better to configure an additional tool schema to ensure the response is formatted to your specifications.

1 Like
16

this may help you and others on your journey.