Realtime API tool calls frequently return server errors

API Bugs
1

I’m working to integrate Realtime API into my mobile application (WebRTC) with some simple tool calls configured around camera control:

Future configureSession({
// params used in instructions,
}) async {
// same params
final sessionConfig = {
“type”: “session.update”,
“event_id”: DateTime.now().millisecondsSinceEpoch.toString(),
“session”: {
“modalities”: [“audio”, “text”],
“instructions”: “omitted for brevity, but in part they describe how the model can control the camera using tools. Includes other context via params”,
“voice”: “alloy”,
“temperature”: 0.6,
“max_response_output_tokens”: “inf”,
“input_audio_format”: “pcm16”,
“output_audio_format”: “pcm16”,
“input_audio_transcription”: {
“model”: “whisper-1”
},
“turn_detection”: {
“type”: “server_vad”,
“threshold”: 0.6,
“prefix_padding_ms”: 300,
“silence_duration_ms”: 500,
“create_response”: true
},
“tools”: [
{
“type”: “function”,
“name”: “startRecording”,
“description”: “Start recording the self-tape camera”,
“parameters”: {
“type”: “object”,
“properties”: {},
“required”:
}
},
{
“type”: “function”,
“name”: “stopRecording”,
“description”: “Stop recording the self-tape camera”,
“parameters”: {
“type”: “object”,
“properties”: {},
“required”:
}
}
],
“tool_choice”: “auto”
}
};

print('📤 Sending session config: ${jsonEncode(sessionConfig)}');
_dataChannel?.send(RTCDataChannelMessage(jsonEncode(sessionConfig)));

}

I’m noticing the audio in, audio out interactions work quite well. When I ask to start or stop recording, however, it clearly recognizes a tool is needed but the response is often:

“The server had an error while processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if the error persists.”

Has anyone else had this issue?

2

I’ve had similar issues as well. I can’t have a consistent conversation with the API. I’ll send one audio buffer, get a response, then when I try to send a second I get this error.

status_details: {
    error: {
      message: 'The server had an error while processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the session ID sess_AmB6LBx6Fg9o8thCrg4Ld in your message.)',
      code: null,
      type: 'server_error'
    },
    type: 'failed'
  }