Realtime API 1.5 - Tone changes per call

API
,
1

Hello Dear Team

I have a report from the engineering team telling me that “With Each call, the tone of the AI Agent changes”

Indeed this is sometimes true.

The Tone is different with each call. NOTE: Keeping the voice the same at all times.

Do you guys encounter the same symptom?

I of course keep this in mind at all times

https://developers.openai.com/cookbook/examples/realtime_prompting_guide

I understand that the Realtime API models are non-deterministic with voice/tone by default. Even with the same system prompt, you’ll get different vocal qualities each session because the model is sampling from a range of plausible voices and deliveries. But is there something I can do?

The guide mentions to lower the temperature.

Does the realtime model support temperature parameter?

Thanks so much
God bless

2

Yes, I’ve seen this as well.
Moreover, I’ve seen sometimes it’ll mispronounce words randomly (this is more rare but still happens).

3

I have seen this too. Some variation between sessions is expected, but I am seeing some particularly frustrating behaviours:

  • The tone/accent of the voice changes mid conversation, and sometimes a female voice suddenly becomes more masculine
  • The tone often changes after a tool call to become more alert and less relaxed. I have had some success prompting something to the effect of “don’t panic after a tool call”, but there is still discontinuity in the tone. For example if the model is appearing humorous and relaxed, after a tool call, it appears formal and stiff like a news reader
4

This also still seems to happen on Realtime 2 in my testing?

5

There is no temperature control parameter or temperature echo return in the realtime session object. It was taken out of the non-beta version of realtime. There is no indication of what the model is being run at - or if there is complete oversight of the need to constrain temperature on voice models to avoid the very symptoms reported here.

Chat Completions has the model gpt-audio-1.5, where you can experiment with lowered temperature and see consistency when going even lower than the default 0.8 of 0.6-1.0 range you previously had, but where as low as “0” can be problematic and can have the AI start generating silence. No 2.0 version there.

6

Hi!

I was looking through past discussions and found this example, which may be worth checking.

In short, the developer noted that exceeding the context window can affect instruction following.

7

Hello, yes this referenced post was from me. I can confirm this is not the case with realtime 1.5 or 2, there is something else wrong.

8

I will raise the with team. Thank you for confirming it!

9

thank you that’s very kind, I can reproduce and provide call ID’s if required. Thanks again.