I have been wondering whether the new logprobs option for chat completions increases counted tokens for billed usage. I.e., when I set n=1 and top_logprobs=5, will it bill me for 1 or 5 tokens? Has anyone already tried that out?
It only bills you for the generated tokens.
After each token the model computes for all possible tokens the probability that token is the next one in the sequence. That’s the computationally expensive step and what costs you (and OpenAI) money. Then the model pulls a sample from all possible tokens weighted by token probability, that’s cheap to the point of basically being free.
Where the logprobs come in is just finding the top five most likely next tokens at each point in the generation and spending those to the return object. That’s also very cheap–basically free.
So, there’s no reason why returning logprobs would increase the cost to you as the user of the API.
2 Likes
Yup - the logprobs parameter does not cost you more. Similar to how setting the logit_bias does not.
Pricing for chat completions is in terms of input and output token as mentioned here
2 Likes