Realtime API interrupts too aggressively on filler words (hmm, ok, etc.)

API Bugs
, , ,
1

Hi everyone,

I’m currently building a real-time voice AI agent using the OpenAI Realtime API (gpt-realtime), and I’m facing a major issue with interruptions.

In natural conversations, users often say filler words like “hmm”, “uh-huh”, “ok”, or “right” just to acknowledge they are listening. However, the Realtime API seems to treat these as actual interruptions.

As a result:

  • The assistant stops speaking immediately

  • It assumes the user wants to take over the conversation

  • This breaks the flow and creates a very unnatural experience

In many cases, the user is not trying to interrupt at all they are just passively acknowledging.

I’ve already experimented with multiple parameters such as:

  • vad_threshold

  • prefix_padding

  • silence_duration

But none of these seem to reliably solve the issue.

Another related problem is that when interruptions happen, the transcript/context handling is not always consistent. The assistant sometimes behaves as if the user heard the full message, even though it was cut mid-way, which leads to confusing or disjointed responses.

What I’m looking for:

  • A way to distinguish between acknowledgment sounds vs real interruptions

  • Better control over interruption sensitivity

  • Best practices for tuning turn detection in real-world conversations

Has anyone found a reliable workaround for this?

Or is there an official recommendation for handling this kind of scenario?

Thanks in advance :folded_hands:

2

Hi! Did you ever manage to resolve this? I’m seeing the same behavior—short acknowledgments like “hmm,” “okay,” or similar responses keep interrupting the agent, which makes the conversation feel much less natural.

3

Solved this by disabling server VAD and using a semantic-turn STT for turn detection, Deepgram Flux in my case. The key isn’t a threshold, it’s that Flux understands meaning, so backchannels like “hmm/ok” don’t register as a turn the way energy VAD forces them to. eot_threshold ~0.5 works well for me.