Hi people, I want to show you an interesting issue I’m having using openai-node on a Remix project.
High level, the server part makes the request to OpenAI api and returned stream is embedded in the response. The client side which made the request using fetch parse the streamed response.
Speaking of code, the server part is this:
const stream = openai.beta.threads.runs
.stream(threadId, {
assistant_id: assistantId,
include: ['step_details.tool_calls[*].file_search.results[*].content'],
})
.on('messageDelta', (message) => {
console.log('messageDelta', JSON.stringify(message));
})
.toReadableStream();
return new Response(stream, {
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
Thread: threadId,
},
});
The client part is
const response = await fetch(localizedChatAPIURL, {
method: 'POST',
body: thePayload,
});
const reader = response.body
.pipeThrough(new TextDecoderStream())
.pipeThrough(new TextLineDecoderStream())
.pipeThrough<AssistantStreamEvent>(new JSONDecoderStream())
.getReader();
The strange issue is that messages appears to be different from server to client.
The output of that console.log on messageDelta, server side is:
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"E","annotations":[]}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"ddy"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":" Mer"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"ck"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"x"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":","}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":" noto"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":" come"}}]}
While the network inspector on the browser side logs these messages:
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"Eddy Merckx, noto","annotations":[]}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"ddy"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":" Mer"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"ck"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"x"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":","}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":" noto"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":" come"}}]}}}
The issue here is that this inconsistency is causing malformations of the first sentences. The example is in italian, I’m sorry, but the beginning of that sentence should be “Eddy Merkx , noto come”. Instead, the client part prints “Eddy Merckx, notoddy Merckx, noto come”.
I noticed the first message on the client side is longer than the first message on the server side so I’m suspecting some buffering happening somewhere.
Do you people have already encounter this issue? Do you have feedbacks about how to handle this and avoid the malformations?
Thank you