Constantly disconnecting after session update with Realtime API

[nlinx]

Every time I connect to the realtime api via websockets and send a session update, it closes the connection and returns a 1000 code.

This is my code

new WebSocket(
            "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-12-17",
            {
                headers: {
                    Authorization: `Bearer ${config.openaiApiKey}`,
                    "OpenAI-Beta": "realtime=v1",
                },
            }
        );

    private setupOpenAIWebSocket(): void {
        this.openAIWs = new WebSocket(
            "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-12-17",
            {
                headers: {
                    Authorization: `Bearer ${config.openaiApiKey}`,
                    "OpenAI-Beta": "realtime=v1",
                },
            }
        );

        this.setupEventHandlers();
    }

    private setupEventHandlers(): void {
        this.openAIWs.on("open", () => {
            this.logger.info(
                {
                    agentId: this.agent.id,
                },
                "Connected to OpenAI Realtime API"
            );
            this.sendInitialSessionUpdate();
        });

        this.openAIWs.on("message", (data: string) =>
            this.handleOpenAIMessage(data)
        );

        this.openAIWs.on("close", (code: number, reason: string) => {
            this.logger.info(
                {
                    agentId: this.agent.id,
                    code,
                    reason,
                },
                "Disconnected from OpenAI Realtime API"
            );
        });

        this.openAIWs.on("error", (error: Error) => {
            this.logger.error(
                {
                    error,
                    agentId: this.agent.id,
                },
                "OpenAI WebSocket error"
            );
        });
    }

It used to work and now it worked half the time; the other half it just closes with a code=1000 after I send the session update.

I also get this error from openai after sending the session update

{
  type: 'error',
  event_id: 'event_AmqwXzqaPM3t6xZ7bhdHm',
  error: {
    type: 'server_error',
    code: null,
    message: 'The server had an error while processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the session ID sess_AmqwWcQVRKxulkoohHV2N in your message.)',
    param: null,
    event_id: null
  }
}

[james.gobert]

I’m having a similar problem running Flask-SocketIO backend. When sending full audio it disconnects after first response.

I then tried using server_vad for real-time chatting (assuming I’d keep the connection open) but I can’t even get a full response back. It keeps disconnecting midway through speaking the first response with a code=1000

Going nuts because I’m not sure if it’s something I’m doing wrong, the API being weird or both.

[hussonsam]

Hello, same issue here, it used to work fine but now i get a code 1000 after the first reply with a function. I cannot keep the connection open. I have noticed this issue for few days, been making many changes to my code but not been able to find a solution.

[louzell]

I am also facing this. Upon sending the first ws message OpenAI always disconnects. Same code as the others, 1000. Here’s what the request, response, and ws messages look like for me.

I’ve also tried just sending a response.create message, and that also exhibits the same behavior (immediate disconnect).

What’s interesting is if I don’t send any message, OpenAI happily keeps the ws connection open.

Request:

Response:

WS messages:

[annshatilo.tt]

Same problem, any fixes found?

[louzell]

Not yet, I’ve been battling it all morning.

What’s really peculiar is if I bounce the connection from client through my backend and then to OpenAI, the connection stays open and all ws messages are delivered just fine. But if I connect from client straight to OpenAI, OpenAI drops the connection as soon as the client sends the first message.

So with this in mind I figured I could look at the headers that are sent from my proxy to OpenAI and compare them with the ones that I send straight from client to OpenAI. And this is where I’m really scratching my head, because I’ve made the headers exactly the same and the connection still drops when going straight from client to OpenAI!

I’m looking at opendiff right now comparing the headers. The same set of headers is being sent up unless I’m really missing something.

So this rules out incorrect request headers causing the issue. Now I’m wondering if it’s some specific opcode on the websocket comm that is different. Maybe the client that I use in the node proxy sends ws pings automatically and Apple’s URLSessionWebSocketTask does not.

But I’m skeptical that it’s ws ping/pongs, because the connection stays open for as long as I don’t send a ws message. E.g. if I connect and wait 10 seconds to send a “response.create” message, then the connection closes in 10 seconds. It always closes on the very first message after OpenAI responds with “session.created”.

I was also thinking it could be a race, so I tried first sending ‘session.update’ before ‘session.created’ is received, and then tried waiting for ‘session.created’ to be received before sending ‘session.update’. Same result in both cases.

[haseeb_1998]

@nlinx

this.openAIWs.on("message", (data: string) =>
            this.handleOpenAIMessage(data)
        );

The issue lies in the fact that you’re passing the entire MessageEvent to OpenAI instead of just the data field within the event.

To fix this, you need to extract and pass the data field from the event. Here’s an example of how you can do it:

for example

this.openAIWs.on("message", (event: string) =>
            this.handleOpenAIMessage(event.data)
        );

This ensures that only the data field, which contains the message content, is passed to openAi.

[aidanallchin]

[FIXED]
I had the same issue with a variety of different strategies, all would fail seemingly without an error message and just the 1000 (OK) response. It turns out that the “error” message is caused by unexpected but not necessarily incorrect content in the websocket stream. I changed two things that fixed this:

1. I had slight errors in the formatting of the messages I was sending from the client. I recommend very carefully checking the expected structure in the OpenAI docs, especially the session config message as well as any manually injected conversation items.

2. I was sending audio information before the websocket was ready to receive it. I fixed this by buffering audio while waiting for the first response.done server message, and then sending an initial message with that content (batched). Keep in mind that if you’re injecting an initial conversation item, the model will respond to both the initial injection as well as your batched audio before being open to natural conversation, so you may also just want to discard that initial audio content prior to receiving the response message.

Hope either of those two ideas help anyone dealing with the same issue!