Wildly different INPUT token counts

API
1

I’m building and testing a new o3 prompt that has a simple text prompt and web search activated (no files or other features used). The user provides a short message as input. The prompt plus message is only around 4000 text characters.

The value of INPUT tokens I get in the logs vary wildly even when using the same user message.
I have seen INPUT tokens vary from around 2k tokens to over 250k (120x!) tokens with almost identical requests. How can this be?

Shouldn’t the input token count be deterministic?

Any help solving this would be greatly appreciated! What am I missing?

p.s.: The only option I see is that the search results are being included as INPUT. This is confusing though as the search action is being shown in the “output” section of the logs. Is this it? Are search tokens considered as INPUT or OUTPUT? Thx!

2

Yes, it is counted as input tokens. Hover your mouse over the tokens in the dashboard for details.

image
image476×282 8.68 KB

Also, before you panic, check your usage property in the response object for details. Much of the input may be cached, which will drastically reduce expected costs.

{'input_tokens': 12898,
 'input_tokens_details': {'cached_tokens': 8320},
 'output_tokens': 628,
 'output_tokens_details': {'reasoning_tokens': 512},
 'total_tokens': 13526}

It is not deterministic because you can’t predict how many searches the reasoning model will make, neither the result size of each web search.

You can define a max_tool_calls parameter, to help constrain a bit the usage (source).

3

Also to note: o3 is tool-happy, and will use the web search with many repeated calls depending on how desperate it is to find information for you.
The internal “chat” grows each time, and each new run after a tool return is a new billed input of all that has been gathered.

On the API, you can limit this with the parameter for the maximum tool calls before an answer is forced: max_tool_calls, sure, but then the AI might not be happy with the results.

4

Thanks guys for the confirmation. Would be nice if the mouse over breakdown for input was separated into prompt and tools to better understand this.

Is there a way in the dashboard logs to see the full raw response object? I can’t see a way to see the raw results.

Thanks also for the tip on the max tool calls. Might try it although I wouldn’t want to impact the results too much for this use case.

5

In the dashboard it is not possible, but for a particular stored response that you still have the ID, you can retrieve the full response object with the API:

response=client.responses.retrieve('resp_123456')
print("usage:",response.usage.to_dict())
print("raw:",response.to_dict())