Language detection check forcing me to waste a lot of tokens

API
1

Hello Dears

Model: Openai API Realtime mini or realtime 1.5

IMPORTANT: It is an existential problem (happened since the beginning with realtime model also - this is not something new. )

Problem Desc:

  1. Caller Calls in one language example English
  2. Language is detected (As English) and turn completes in the language English.
  3. Caller Speaks again in the SAME language as before (English)
  4. Realtime model detects another language. (very unpredictable) and turns in a different language.

Solution (The solution We were forced to do): (The only way we could get predictability)

Call the Language Detection TOOL {set_detected_language} with EACH TURN.
And now we have a good release and it is stable. Great. We released and it’s in production. So It is working.

====

But at this stage, I am not happy with just stability or just “its working”.

I made a count of a standard call (4 min, couple of turns, vector storage question and answers") - nothing fancy and purely realistic. Imagine an example of a hotel or booking.com with questions like “how much is the room, do you have a pool, do you have breakfast”, very simple utterances.

And counted how many tokens we are wasting just by firing this tool. The result is shocking.

Every time the caller spoke, our code injected a system message:

“Update the participant’s current language before continuing.” – which forces a full response cycle just to call set_detected_language(“English”) and get back “Language set to: English”.

For this standard call, it happened 13 times in this call, always returning the same result.

Each cycle costed:

  • 1 response_created +
  • function_call round-trip +
  • function_call_output +
  • another response_created.

That is roughly 1,400 tokens of overhead per turn just for this one pattern.

Across 13 turns I consumed: 18,200 tokens.

18k out of 32k is more than 50%!!! how can I sustain this behavior?

The design intent is correct — detect language once. But I am forced to re-detects on every single turn regardless of whether the language has changed or was already known otherwise the realtime model WILL drift and start to detect languages without control.

And if I remove this? What will happen?

You can for example speak PERFECT English and if the caller’s name happens to be of Arabic descent example “Amir”, then realtime switches to Arabic. Or if the caller is Sofia, you would get Russian. Sarvesh speaking perfect English, might get hindu back. Take my word for it. Its a problem. And yes, I tried at length in the prompt telling the model “DO NOT CHANGE BASED ON ONE WORD OR ONE NAME OR ONE UTTERANCE” etc etc. Useless.

Or it might catch some dialect - example Perfect english, with a little Italian accent - Will switch to Italian or Spanish right off the bat.

Are there ways to overcome this, or plans to improve the language detection in the future?

Thanks so much guys and keep up the great work!!

God bless

2

You can instruct the model to call the language detection tool only once for the entire request, and then use that result to set the transcription language.

3

Hi dear @sps

Thanks for your reply - Yes and we do use this approach.

And then how can I support language switching? Example

  1. Customer is in the US, and greets you in English.
  2. Caller is Spanish and starts in Spanish.
  3. It’s currently very convenient when the realtime turn gets to be in Spanish. Excellent negotiation.

How can I have support for both?

4

Hi @nbo2,

I understand the dilemma, but if the goal is to support language switching, it is necessary to check the language on each turn of the conversation.

From what I understand, the issue is that setting the conversation language does not work well enough for your use case. Is that correct?

You may already be aware of this, but just in case, the realtime prompting guide may contain additional information that could help improve the situation.
Maybe guiding with an example can help?: ‘If a user is communicating in English, the conversation should remain in English regardless of their name or accent.’

5

hi dear @VeitB

Agreed - As long as this is the only way, then I am ok.

Yes this is the issue - setting the conversation language might switch when you don’t want it based on a word.

Thanks for the realtime prompting guide - I have these guidelines always close by. thanks so much God bless.

6

Interestingly enough, the issue the OP raised here has given me a whole new take on handling the Ai empowered robo callers my nation has been plauged by recently.

:clinking_beer_mugs:

Cheers, OP.