Issue with Chunk Streaming in ASP.NET Core using GPT-4 API

API
, ,
1

Hello OpenAI Community,

I’ve been using the GPT-4 (gpt-4-0125-preview model) chat completion stream API in an ASP . NET Core application for a few months now. My setup involves streaming from my client-side service to the OpenAI service. Everything was functioning smoothly until recently when I encountered a peculiar issue without making any new deployments to the server.

The problem is with the streamed object chunks, which are being returned corrupted to my client. An example of the returned object is as follows:

}{"id":"89dd2ae0-04db-4555-84d4-5ab7da6a952b","content":null
next chunk
‘}’.
Notably, the end of one object and the start of the next object are split across chunks.

Here is the relevant client-side code:

javascriptCopy code

async function* streamAsyncIterable(stream: any) {
    const reader = stream.getReader();
    try {
        while (true) {
            const { done, value } = await reader.read();
            if (done) {
                return;
            }
            yield value;
        }
    } finally {
        reader.releaseLock();
    }
}

if (!res.body.getReader) {
    const body = res.body;
    if (!body.on || !body.read) {
        throw "";
    }
    body.on("readable", () => {
        let chunk;
        while (null !== (chunk = body.read())) {
            feed(chunk.toString());
        }
    });
} else {
    for await (const chunk of streamAsyncIterable(res.body)) {
        const str = new TextDecoder().decode(chunk);
        
        let data = processStringData(str);
        data.forEach((item:any) => {
            try {
                onMessage(item);
                feed(JSON.stringify(item));
            } catch (error) {}
        });
    }
}

Interestingly, when I run the program locally, everything works fine without any issues. However, when deployed on my IIS server, the problems start to occur. I have tested this on several servers, but the issue persists.

Any guidance or suggestions from the community would be greatly appreciated. Thank you!