Regarding OpenAI api pricing.
I want to know that if I provide max_tokens of 1000 in API request, but the total tokens (prompt + response) usage is 300. Will I be billed for 300 tokens or 1000?
I’m asking for both chat and completion based models.
You can specify max_tokens without fear to your pocketbook. You are only charged for the tokens generated, even if you reserve context length space for more.
While the chat models are well trained in ending their answers, keep in mind that completion models aren’t, and will often go off on unrelated token rants or will repeat until they run out of room, so it is a good idea to set max_tokens in your API call for just a bit more than the response you expect, especially when venturing into unknown territory.
1 Like