Incomplete Words in Streaming

Developer Community,
I want to inform you about an issue we have detected in the stream API of the models provided by OpenAI.
This issue occurs specifically when generating responses in real time via streaming and affects the integrity of the responses by leaving incomplete letters in the results.
Furthermore, we have observed that when using this function, errors occur in the token count, and even in some cases, when invoking models, they do not correctly reflect the requested functions. These errors are solved by disabling the stream function.
This has resulted in resource consumption that does not correspond to our normal operations. A plus happens with the 3.5 turbo models.

1 Like

Take a look at this topic. It should help you to understand and resolve the issues with the token count and the incomplete responses.

In short: take a look at the stop reasons for each chunk and count the tokens using tiktokken.
I recall that this was explained in the docs but I can’t find it any more.

Hope this helps.

Everything is already resolved, without using stream everything is correct, when activating stream everything goes out of control, consumption, invocation of wrong models. and the problem is not only with the API, it happens the same in the plus accounts.

Thank you very much. I just saw it today

1 Like