Issues with realtime turn-taking

API Bugs
, ,
1

I’ve observed two issues with realtime turn-taking that I’d like to report.

  1. The interrupt_response configuration doesn’t seem to have any effect. I’ve tried setting it to False, but the LLM will still stop talking if I start talking over it. Am I misunderstanding the purpose of this configuration? I expected the LLM to finish speaking its whole response with this config set to False. I’ve observed this using both server and semantic VAD.

  2. Semantic VAD often doesn’t pick up on short utterances by the caller, like “yeah” or “sure”. I noticed this problem because I have a tool call that sends an SMS that I want the LLM to make sure the caller explicitly opts into. So the LLM says something like, “Do you want me to send that text to your phone?” If the caller just says “yeah”, no input is registered. There’s no input_audio_buffer.speech_started event, the caller’s turn isn’t registered, so then there’s silence on the call until the user says something more obvious like, “Yes, send the text.” Server VAD does a good job of picking up on these short utterances so I’ve switched back to using it for now. When I was using semantic VAD I had eagerness set to high.

2

We’re observing the same behavior and want to confirm whether this is a bug or a misunderstanding of interrupt_response.

We’d like the AI to finish its spoken response even if the caller starts speaking, so we’ve set it to false.

However, this parameter seems to do nothing in practice.

As soon as input_audio_buffer.speech_started comes in, output_audio_buffer.cleared happens, which immediately cuts off the assistant’s audio mid-sentence.

This happens regardless of whether interrupt_response is set to true or false.

Curious if anyone has found a workaround or seen different behavior.

3

Regarding the first point, I have the same problem .. the way I circumvented it was by setting turn_detection to null for the initial responses where i don’t want the model to be interrupted and then sendind a session.update to set the entire turn_detection object with my vad settings

4

@Sean-Der Do you know what’s going on here with the interrupt_response configuration?

I agree with @achill that being able to make certain responses non-interruptable would be really useful. A welcome message is a perfect example of one that we wouldn’t want being cut off. Changing the turn detection settings just to accomplish this is a bit cumbersome though, so would it be possible to allow new field on the response.create client event to make just the single resulting response un-interruptable?

6

We can consistently reproduce the assistant’s audio being cut off mid-sentence by the server clearing the WebRTC output audio buffer on VAD speech detection, even though our session explicitly sets interrupt_response: false. Adding event-level telemetry to this thread because the API reference wording suggests this may not be intended behavior.

Setup

  • Model: gpt-realtime-2.1, browser WebRTC transport (GA /v1/realtime/calls)
  • turn_detection: { type: "semantic_vad", create_response: false, interrupt_response: false }
  • noise_reduction: { type: "far_field" }, input transcription gpt-4o-mini-transcribe (language: "ko")
  • Output modality: audio. Client does not send output_audio_buffer.clear or response.cancel in these traces (we log all client control sends).

Expected

The API reference for interrupt_response says: if true the response is cancelled on VAD start, “otherwise it will continue until complete.” We therefore expected the in-progress response — and its already-generated audio draining from the server buffer — to keep playing through short non-speech noises (coughs), with turn creation already disabled by create_response: false.

Actual

On every input_audio_buffer.speech_started that fires while assistant audio is playing, the server immediately emits output_audio_buffer.cleared and the remaining audio is permanently dropped. This happens in two situations:

  1. while the response is still generating (response.done not yet received), and
  2. after response.done, while the buffer is still draining (long answers).

The triggering “speech” is frequently a cough or the onset of the assistant’s own audio leaking into the mic — the subsequent conversation.item.input_audio_transcription.completed arrives with an empty transcript or a filler, i.e. no real user turn. interrupt_response: false makes no difference. A conversation.item.truncated event accompanies the clear, so the server is running its full interruption/truncation path despite the flag.

Sample timeline (from our telemetry, Asia/Seoul):

13:08:53  user question (real turn, transcription ok)
13:08:56  response A: forced function call chain (create_response:false, client-driven)
13:09:07  response.done (answer B generated; audio draining, audibly playing)
13:09:13  input_audio_buffer.speech_started        <- cough while answer B audible
13:09:13  output_audio_buffer.cleared (response B) <- remaining audio dropped
13:09:13~ conversation.item.truncated (response B)
13:09:16  input transcription completed: filler only — not a real turn
          => answer B is dead mid-sentence; nothing recoverable client-side

Same pattern reproduced across many calls, on 55–65s answers cut 7–17s in.

Why this matters

With WebRTC the server owns the buffer, so once cleared the audio is unrecoverable by the client — a single cough kills a long answer. We’ve had to build an out-of-band “re-speak from where it cut off” workaround (conversation: "none" response), which works but adds a 2–3s gap and token cost.

Questions

  1. Is the buffer clear + truncation on speech_started supposed to be gated by interrupt_response: false? The reference wording (“will continue until complete”) suggests yes, in which case this is a bug.
  2. If it is intended, is there any supported way to keep semantic VAD on (we need speech_started/transcription for turn-taking) while disabling server-side output clearing?

Happy to provide more event traces.