Realtime transcription Problem - gpt-live-transcribe - no final events received

API
,
1

Hello Dear team

We are following the documentation here:

we tried the models:

  • gpt-live-transcribe (new one, the one we want to use)
  • gpt-realtime-whisper

In both cases we have problems.

We expect to receive both deltas and final events:

ws.on("message", (data) => {
  const event = JSON.parse(data);

  if (event.type === "conversation.item.input_audio_transcription.delta") {
    process.stdout.write(event.delta);
  }

  if (event.type === "conversation.item.input_audio_transcription.completed") {
    console.log("\nFinal transcript:", event.transcript);
  }
});

However, we are not receiving final events:

We realized that the session can be configured only with turn_detection turned off.

{
  "type": "session.update",
  "session": {
    "type": "transcription",
    "audio": {
      "input": {
        "format": {
          "type": "audio/pcm",
          "rate": 24000
        },
        "transcription": {
          "model": "gpt-live-transcribe"
        },
        "turn_detection": null
      }
    }
  }
}

If we do this, we receive only deltas but not final events

After this we follow the documentation to enable turn detection as reported here:

specifically this disclaimer

To let the server detect and commit turn boundaries, configure voice activity detection instead.

{
  "type": "session.update",
  "session": {
    "type": "transcription",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "transcription": { "model": "gpt-live-transcribe" },
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.8,
          "prefix_padding_ms": 500,
          "silence_duration_ms": 700
        }
      }
    }
  }
}

But then we get this error:

{
  "message": "Turn detection is not supported for this transcription model.",
  "type": "invalid_request_error",
  "code": "invalid_value",
  "param": "session.audio.input.turn_detection"
}

Can you please guide us and tell us what we are doing wrong?

Thanks so much guys

God bless

2

@sps

Dear Sps can you please check this because we are stuck

Thanks so so much

3

Hi @nbo2,

Thanks for reaching out. For gpt-realtime-whisper transcription sessions, turn detection must be set to null; VAD is not supported.

When turn_detection is set to null, i.e., when automatic turn detection is turned off, you can commit the buffer when you want to finish an audio turn:

ws.send(
  JSON.stringify({
    type: "input_audio_buffer.commit",
  })
);

You will then receive a completion event containing the final transcript for the committed item:

{
  "type": "conversation.item.input_audio_transcription.completed",
  "item_id": "item_003",
  "content_index": 0,
  "transcript": "Hello, how are you?"
}

Regarding the errors you’re receiving when using automatic turn detection with gpt-live-transcribe, I’m going to flag this with the team at OpenAI so they can confirm whether gpt-live-transcribe supports automatic turn detection.

4

Thanks so much @sps Much appreciated.. We will check and keep you posted. In the meantime, please let us know what the team tell you.

God bless

5

Hi Dears

Do you have any updates on this?

If openai api is not going to tell me when the end of turn occurs, I am forced to develop my own VAD.

Can you please let me know because we find it strange that you do not provide this information?

6

Confirmed: Just like gpt-realtime-whisper, the model gpt-live-transcribe supports only "turn_detection": null.