Difference token count between gpt-4 and gpt-3.5 (also gpt-4-1106-preview) , does anyone know the reason?

API
, , ,
1

I tried to test the below request to OpenAI api.
It returns tokens with 35.
When I changed the model to gpt-3.5-turbo or gpt-4-1106-preview, it returns 34.

Anyone know the calculation logic? What is the difference between gpt4 and gpt-3.5-turbo?

{
	"model": "gpt-4",
	"messages": [
		{
			"role": "assistant",
			"content": null,
			"tool_calls": [
				{
					"id": "call_Id8ycVMsW8gdsf7kSXfgAcf1",
					"type": "function",
					"function": {
						"name": "get_current_weather",
						"arguments": "{\n  \"location\": \"Boston, MA\"\n}"
					}
				}
			]
		},
		{
			"role": "tool",
			"tool_call_id": "call_Id8ycVMsW8gdsf7kSXfgAcf1",
			"name": "get_current_weather",
			"content": "29 degree celcius"
		}
	]
}
2

After investigated this issue for few days, I found that they are the same token count when using gpt-4, gpt-3.5 and other popular models.

It is cased by the arguments in tool_calls.

When we send tools to OpenAI, we will get below message:

"tool_calls": [{
    "id": "call_h921N3VXwHw0RI7fgn6USsKS",
    "type": "function",
    "function": {
        "name": "get_weather",
        "arguments": "{\n\"location\":\"Shanghai\",\n\"unit\":\"celsius\"\n}"
    }
}

Sometime it returns the function like:

  "function": {
        "name": "get_weather",
        "arguments": "{\"location\":\"Shanghai\",    \"unit\":\"celsius\"}"
    }

The difference between them is obviously. The space, tab space or newline char in arguments.

After test, I get the conclusion. OpenAI calculates arguments’s token count without space, tab space but newline char. Take an example as below.

 "function": {
        "name": "get_weather",
        "arguments": "{\n\"location\":\"Shanghai\",\n\"unit\":\"celsius\"\n}"
    }

I’ve fixed in version 3.7.20240130 of openai-java

Please free figure out the wrong idea about me if you find.