2 JSON objects returned when using function calling and JSON mode

API Bugs
, ,
1

With gpt-3.5-turbo-1106 when tools are passed to the chat completions API with JSON mode the model will output 2 JSON objects like the following :

{
  "message": "Hello! How can I assist you today?"
}
{
  "message": "Hello! How can I assist you today?"
}

or

{
  "current_temperature": "67°F",
  "average_temperature": "On average, the warmest months in Los Angeles are July through September, with average temperatures ranging from the mid-70s to mid-80s (°F). January through March are the coolest months, with average temperatures ranging from the low 50s to mid-60s (°F)."
}

{
  "city": "Los Angeles",
  "current_temperature": "67°F"
}

If the tools are not passed to the API the model will output a single JSON object which is what I want.

gpt-4-1106-preview doesn’t have this issue and works as expected.

Could it be an issue with the prompt if so how can I fix it.

I’ve tried giving it an example which worked but was inconsistent and the model would leak the example :

{"message": "Hello! How can I assist you today?"}
{"message": "Your response goes here"}

When used on a larger system prompt with 6 examples the model would always return 2 JSON objects, often times different from each other.

This problem could probably be an issue with the prompt which I tried to fix but gpt-3.5-turbo-1106 is really stubborn and doesn’t want to work, is there any guidlines for prompting gpt-3.5-turbo-1106 in JSON mode to achieve consistent results ?

This is an example code that I’m using.

from openai import OpenAI
from openai.types.chat import ChatCompletionToolParam
from openai.types.shared_params import FunctionDefinition
client = OpenAI()

function = FunctionDefinition({'name': 'duckduckgo_search', 'description': 'A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.',
                                         'parameters': {'type': 'object', 'properties': {'query': {'description': 'search query to look up', 'type': 'string'}}, 'required': ['query']}})
toolc = ChatCompletionToolParam({'type': 'function', 'function': function})
print(toolc)
completion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that returns json of your response in a message key",
        },
        {
            "role": "user",
            "content": "hello",
        }
    ],
    model="gpt-3.5-turbo-1106",
    #model="gpt-4-1106-preview",
    response_format={"type": "json_object"},
    tools=[toolc]
)
print(completion.choices[0].message.content)
2

I see two different issues here: (a) why the tool is not used, and (b) why the JSON is duplicating itself.

For the first issue, if a tool is used by GPT, GPT will return the arguments to the tool as a JSON. This is documented here: https://platform.openai.com/docs/guides/function-calling.

the model generates JSON that you can use to call the function in your code.

Also, my understanding is that you don’t need to turn on JSON mode for the tool calling to return a json.

I tried running the code snippet you provided and the tool duckduckgo_search is not used since the input to the llm is simply “hello”. The LLM has rightfully not triggered this tool since it’s not relevant.

Instead, I did the following

completion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that returns json of your response in a message key",
        },
        {
            "role": "user",
            "content": "what is the temperature in Singapore",
        }
    ],
    model="gpt-3.5-turbo-1106",
    response_format={"type": "json_object"},
    tools=[toolc]
)
print(completion.choices[0].message.tool_calls[0])

And got this

{"query":"current temperature in Singapore"}

Note how the print statement is different since you access the arguments for the function call differently.

For the second issue, I think it’s because your system prompt is ambiguous. Do you mean that you want the LLM to always return a JSON with the key message?

Instead, I may try something like this:

You are a helpful assistant that returns json of your response in a message key

If I change the system message, the “Hello! How can I assist you today” json is not repeated.

Lastly, you may find this package instructor for structuring LLM outputs to be helpful - it uses pydantic under the hood.

3

They’re same correct ?

And there’s no issue with the tools they don’t have to work for this example.

4

I don’t really want the tools to be used in this example, since I’m trying to understand the source of my problems.

I used this prompt with an example to get a parsable JSON :

You are a helpful assistant that returns json of your response in a message key like this {\"message\": \"your response goes here\"}

However the model leaked the example and outputted 2 JSON objects as I stated in my initial post .

{"message": "Hello! How can I assist you today?"}
{"message": "Your response goes here"}

Though this only happened once, I want to understand why ? and would it happen again randomly ?

What seemed odd to me is that gpt-3.5-turbo-1106 sends 2 JSON objects when tools are passed to the API.

completion = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that returns json of your response in a message key",
        },
        {
            "role": "user",
            "content": "My name is hench",
        }
    ],
    model="gpt-3.5-turbo-1106",
    #model="gpt-4-1106-preview",
    response_format={"type": "json_object"},
    tools=[toolc]
)

Outputs the following but sometimes outputs JSON object :

{
  "message": "Hello Hench, how can I assist you today?"
}
{
  "message": "Hello Hench, how can I assist you today?"
}

when you comment the tools=[toolc] the model would always output a single JSON object, and THIS IS THE ISSUE

As you can see the model isn’t consistent with it’s response, and passing tools make it even more so, This is what I’m trying to understand with a larger system prompt with more details on how to generate the JSON and even with 10 JSON output examples the model still fails to output a single JSON object.

5

Ah sorry, I pasted the wrong thing. meant a system message like this:

You are a helpful assistant. The format of your reply is a json which has only one key called message which contains the reply.

EXAMPLE
INPUT: Happy New Year
OUTPUT: {‘message’: Happy New Year to you too!}",

As to why this is happening, because we’re using GPT 3.5, maybe it’s necessary to be extra clear in the prompts.

6

It’s fine this makes more sense, however I tested it and out of the 7 runs one of them returned 2 JSON objects:

{
  "message": "Nice to meet you, Hench!"
}
{ "message": "Nice to meet you, Hench!" }

And it seems that the model alternates between the format of the response :
:

{
  "message": "Hello Hench! How can I assist you today?"
}

or

{"message": "Hello Hench! How can I assist you today?"}

Is it safe to assume that this is due to model capabilities ?

I had no success making the model return 2 JSON objects when the tools are not passed to the model, and this is what this topic is about, when tools are passed to the model with json mode he would output invalid JSON since 2 JSON objects are not a valid JSON.