@philippeWander - a couple of observations I have made that might help:
Assumption: Python implementation.
- Regarding tool calls - as you know, when the websocket connection is opened, OpenAI creates the session and we have to send a session.update. Include the tools in that session.update event.
- When the function call happens, you will receive a bunch of response.function_call_arguments.delta events
- wait for the response.function_call_arguments.done event to arrive
- then you need to send a conversation.item.create event like so:
conversation_item = {
"type": "conversation.item.create",
"previous_item_id": None, # You can set this appropriately
"item": {
"id": f"msg_{call_id}",
"type": "function_call_output",
"status": "completed",
"role": "system",
"call_id": call_id,
"content": {
"call_id": call_id,
"name": call_name,
"arguments": call_arguments,
"output": output_message
}
}
}
- then the response.done event will contain the same function call_id showing status as completed.
Watch for an error event from OpenAI like so:
Received event from OpenAI: {'type': 'error', 'event_id': 'event_ANaEFz6z4X38KZNpneqvk', 'error': {'type': 'invalid_request_error', 'code': 'missing_required_parameter', 'message': "Missing required parameter: 'item.call_id'.", 'param': 'item.call_id', 'event_id': None}}
If this error is not fixed, then after the function_call is made, the AI will stop speaking and the only way to trigger a response is via “are you there” from the user or something similar.
Additional observations:
- I suspect your observation regarding VAD via session_update not working is correct, although I have not been able to conclusively test and confirm this.
- I am seeing cancelled response events with “turn_detected” flags which tells me that VAD is working.
- Do you have any such outputs you could share? It should look something like this:
Received event from OpenAI: {'type': 'response.done', 'event_id': 'event_ANaErQa1DdjAms2gXkcMN', 'response': {'object': 'realtime.response', 'id': 'resp_ANaErOidD7tYLMIq5pzIH', 'status': 'cancelled', 'status_details': {'type': 'cancelled', 'reason': 'turn_detected'}, 'output': [], 'usage': {'total_tokens': 0, 'input_tokens': 0, 'output_tokens': 0, 'input_token_details': {'cached_tokens': 0, 'text_tokens': 0, 'audio_tokens': 0}, 'output_token_details': {'text_tokens': 0, 'audio_tokens': 0}}}}
Hope this helps mate… !