Hi everyone,
I’m very new to working with OpenAI and I’m stuck on the following problem: Currently, we are using getChatCompletions() to get the whole response along with all the corresponding data. Now, we want to stream the data. I’ve already managed to stream the content using getChatCompletionsStream(), but when I check the usage, it shows as null in any of the completions.
Is there any way to get the token usage while still using the OpenAIClient?
Thank you in advance, and please excuse me if my English isn’t perfect.
2 Likes
Your english is great. It’s nice to see someone give the effort rather than LLM-strap it up.
Considering that you are mentioning specific types and functions I am assuming you are using some sort of third-party library (OpenAI doesn’t maintain a Java client library). First thing to do is check to ensure that it’s actively maintained. OpenAI is constantly releasing new features that really require the maintainers to be diligent and active.
With this confirmation, there is now a new stream_options object with a boolean include_usage property to send in your request to get the usage information near the end of the stream:
stream_options
object or null
Optional
Defaults to null
Options for streaming response. Only set this when you set stream: true.
include_usage
boolean
Optional
If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array. All other chunks will also include a usage field, but with a null value.
https://platform.openai.com/docs/api-reference/chat/create#chat-create-stream_options
3 Likes
Thanks for your answer. I forgot to check what library was used. It is from Azure. I guess I have to look at their documentation but thank you very much for the tip.