We are using 4 and 4o for a bot that streams responses. As shown below in the response, the response from GPT API has numerous typos. We have tried a low temp of 0.2 and high temp of 0.7 without any impact.
I’m sorry, Andy I’m really reluctant to to dealership. I spent a bit of time during my last visit and it’s frustrating that this completed then. Can explain exactly what needs be done and why wasn’t taken care of my previous visit? , are there any alternative that don’t require me come back in?
We are having the same issue here. And this is specially critical for use cases (such as ours) that require JSON output, as it’ll never be able to properly format the output. For now, we’ve changed to Google’s Gemni API.
I am suspecting that OpenAI documentation is lacking. We may not be processing tokens properly. I have asked my engineering team to look into it. My assumption is that stream happens via UTF and then we convert it into letters and words. Don’t know for sure. But if that’s the case we may need ti set buffer
I also thought this could be the case, at first.
But for us, it apparently started all of a sudden, yesterday. Before that we never had such problem.
I’ve personally investigated this issue, and the chunks sent by the stream are poorly formatted, effectively being an error from the API.
We already had a “connector” for that, so that switching was not a struggle.
But if I’m not mistaken, recently google also announced an OpenAI-compatible api - that is, only by changing the URL / api keys to google’s, it’ll work.
We’re not using that, though, so I’m not sure.
By the way, what is the tech stack you’re using there?
I am not an Engineer so dont know what Tech Stack (Deno) mean. If you can explain in 1-2 sentences I can get you the answer from my engineering team. We are on Azure/Node application.
Oh, you’ve answered already. Deno is a Javascript runtime, similar to Node.js that you are using. With the tech stack question I basically wanted to know the programming language you are using for the application, and / or if you are also using any ‘libs’ (lib stands for 'library") on top of that to interact with the AI APIs (like, node.js has an openai lib that facilitates communicating with their api - we’re also not using it, though. There are other famous ones, specially langchain)
Turns out we’ve worked around the issue. In our case, the objects in the stream were being cut in the middle, and then the remainder of the object would come in the next chunk (even tho it’s the same object).
We created a buffer for the case an object from streamed chunk cannot be properly parsed, wait for the next chunk, and join these partial chunks before handling them.
Up until yesterday we’ve never had this issue. Something changed in OpenAi side.
It’s a bit more complex, and we’ve never experienced before either in OpenAI API, or other providers as well, but seems to be working.