Hello, I’ve been trying to retrieve the Usage information when I stream chat completions using the Azure OpenAI client. I used this post from the OpenAI changelog for guidance:
May 6th, 2024
https://platform.openai.com/docs/changelog/may-6th-2024)
In the Azure OpenAI Client, StreamOptions is an internal field in the ChatCompletionOptions object. By default, StreamOptions.IncludeUsage is set to true (see pic below).
Here is the issue I am having:
Issue: When calling the CompleteChatStreamingAsync() function of the ChatClient class, the StreamOptions field of the passed in ChatCompletionOptions object reference, which by default, has the StreamOptions.IncludeUsage field set to true, is unexpectedly cleared out, and is set to null.
The pictures below illustrate the value of the StreamOptions field before and after the function call (see red stars in pics):
Before:
After:
Here is my initialization of the ChatCompletionOptions object:
var chatCompletionOptions = new ChatCompletionOptions()
{
Temperature = (float)0.8,
FrequencyPenalty = 0,
PresencePenalty = 0
};
I do not touch this object between the time of initialization and the time I pass it to the above function. I have also checked to make sure I am indeed not receiving any Usage info from the StreamingChatCompletion object that is returned from CompleteChatStreamingAsync() . According to the documentation, the last chunk of the stream should contain usage info, but that is not case for me.
Am I missing something (is this intended behavior?)?
Additional Info:
Azure.AI.OpenAI nuget package version: 2.0.0-beta.2
.NET version: .NET 7.0
Let me know if you need any more info
Thanks,
Seb