I absolutely love this feature and just testing it for integration into my app.
Sometimes though, no response is coming back. I talk, get all events, transcription is coming, but no audio response.
Anyone experiences this as well?
Basically just using the web based code example:
...
...
const baseUrl = ".......api.openai.com/v1/realtime";
const model = "gpt-4o-realtime-preview-2024-12-17";
const sdpResponse = await fetch(`${baseUrl}?model=${model}`, {
method: "POST",
body: offer.sdp,
headers: {
Authorization: `Bearer ${EPHEMERAL_KEY}`,
"Content-Type": "application/sdp"
},
});
const sdp = await sdpResponse.text();
console.log("sdpResponse.text()", sdp);
const answer = {
type: "answer",
sdp: sdp,
};
await pc.setRemoteDescription(answer);
Update:
I am basically using this code:
But with model: “gpt-4o-mini-realtime-preview-2024-12-17”;
Also using same server code to get the token.
And I am updating the session to support transcriptions:
dc.addEventListener("open", () => {
const sessionUpdate = {
type: "session.update",
session: {
turn_detection: { type: "server_vad" }, // Sprachaktivitätserkennung
input_audio_transcription: {
model: "whisper-1", // Aktiviert die Transkription
},
modalities: ["text", "audio"], // UnterstĂĽtzt Text- und Audio-Modi
},
};
// Send update to get Transkriptions
dc.send(JSON.stringify(sessionUpdate));
});
Instructions prompt can be quite long, but as I said, sometimes it is working and suddenly stops responding, which is why I have some doubts in releasing it to my users.
I am logging all events using dc.addEventListener("message"
I receive these:
input_audio_buffer.speech_started
input_audio_buffer.speech_stopped
input_audio_buffer.committed
conversation.item.created
response.create
response.done
conversation.item.input_audio_transcription.completed (incl the transcript of my voice)
and I have this:
dc.addEventListener("error", (err) => {
console.error("!!! Data Channel Error:", err);
});
Never gets logged.
Just randomly no audio response played. And I dont get any errors and dont how how to better debug it
Network Conditions: perfect.
Any ideas how I could debug this more sophisticated?