Realtime Transcription Mismatch and gpt 4o transcribe latest

API
, ,
1

I have seen in the new realtime api reference, under transcription object of session.update event, they have mentioned that a new STT Model `gpt-4o-transcribe-latest` is supported but for some reason when i use it in the new GA session, it doesn’t work? It doesnt give the conversation.input_audio_transcription.delta at all. When i revert back to gpt-4o-transcibe it works, not sure what’s happening with oai

Also sometimes the transcripts coming in from realtime doesn’t align with the audio completely, there are instances where

  • the transcript is missing some phrases but in audio its available and the other way around too
  • also the events we are receiving are not always correct and sequential for some reason, there is some mismatch in the events being sent (timing wise), which makes it hard to align the audio with the text
2

below see example of session.update. With this setup:

  1. response.output_audio.delta and response.output_audio.done get you audio stream from the AI

  2. response.output_audio_transcript.done gets you the text transcript of what the AI said

  3. conversation.item.input_audio_transcription.completed gets you the text transcript of what the user said

session_update:

        session_update_message = {
            "type": "session.update",
            "session": {
                "type": "realtime",
                "model": "gpt-realtime",
                "audio": {
                    "input": {
                        "format": {          
                            "type": "audio/pcm",
                            "rate": 24000
                        },
                        "noise_reduction": {"type":"far_field"},
                        "transcription": {
                            "model": ""gpt-4o-mini-transcribe"
                        },  
                        "turn_detection": {
                            "create_response": True,
                            "interrupt_response": False,
                            "prefix_padding_ms": 300,
                            "silence_duration_ms": 750,
                            "threshold": 0.5,
                            "type": "server_vad"
                            }
                        },
                    "output": {
                        "format": {
                            "type": "audio/pcm",
                            "rate": 24000
                        },
                        "speed":1,
                        "voice": "coral",
                    }
                },
                "instructions": "YOUR PROMPT HERE",
                "max_output_tokens": 1024,
                "output_modalities": ["audio"],
                "tool_choice": "auto",
                "tools":[TOOLS HERE IF ANY],
                "tracing": None,
                "truncation":"auto"
            }
        }
3

thanks for sharing, but may I ask you if this is answering Madhuns question?

We have the same experience on our end. The audio2audio experience is pretty good, but as soon as we compare the users audio input with the transcription, it is getting worse. On top of Madhuns issues we have sometimes different languages between audio and text…

4

@Co_Brainers I’m partially answering Madhun’s question… how I get the transcripts. Specific things I am not addressing:

  1. use of “…-latest” model… I don’t like those since it is highly volatile… basically they are just a pointer to the real model so why not choose that one explicitly? So I would not do that in general and I have no opinion on how to make it work

  2. transcription models don’t match realtime models: I do see this in practice. The realtime models work directly off of audio tokens meaning they don’t need text… the fact that you’re getting text is just a convenience and there’s no guarantee that the content as understood by realtime will match the text as transcribed by a different model. If this is a big problem, I would start by using full models (not -mini). Ultimately you may have to chain the pipeline microphone→STT→llm→TTS→speaker so that text is truth… but latency will be a challenge.

1 Like
5

I have also noticed that the audio emitted by the realtime model often differs wildly from the transcribed text that is sent.

6

thank you very much for the elaborated answer. I understand both point you have made. We had a working pipeline of google ASR → gpt4o → google TTS, which was pretty good so far, no latency problems either. The disadvantages recognizing named entities, e.g. names, email adresses were drinving us to try s2s and the results are much better - downsite is the divergence between voice/text and volatility in language detection.

We have only the need for realtime s2s, so I am currently thinging of changing to an post processing approach to get the transcription work done .. br andre