Using callback functions with the Realtime api

API
1

NOTE: I renamed the title and updated the post below to be more precise, so ignore the first several replies up until the last one I deleted. Unfortunately I couldn’t delete this post and start a new one.

This concerns the documentaton on function callings:

https://platform.openai.com/docs/guides/function-calling

which I found to be poorly written. For the Realtime api, there was no indication of the entire flow from the moment an audio session begins until an audiio response is returned and a callback function is required during the processing.

There is no indication where the code intercepting the callback from the Realtime api actually resides. Judging from the sample code for callback functions, which is written for Node, I initally assumed that the interception of the Realtime api response occurred on the developer’s backend since that is where most Node apps are run. But this doesn’t make sense since that code has no access to the stream between the client and the Realtime api. So this means that the Realtime would probably respond back to the client when it needs to call an api. The client would then just call the developer’s backend api, get the result and send it back to the Realtime api. While this makes sense, it doesn’t indicate where the actual registration of the callbacks occur. I can only assume that they take place on the client. But this would mean that they would have to be registered before the actual audio conversation even starts. Or is the client code suppose to just send a list of all the callbacks it supports with every prompt? That would sound like a bad idea.

2

A “function call” is just a different style of output that the AI can write.

Take this example conversation:

{user: what is the weather in London UK}
{assistant: I’m sorry, but I don’t have realtime access to weather}

When you make an API call, you also can send a specification of a function, which is code that you have written and will run to satisfy an AI’s need for more knowledge or to help the AI take action.

Now imagine I’ve given the AI the same question, but it has a developer’s tool available:

{user: what is the weather in London UK}
{assistant: tool_call=weather({“city”: "London, UK})}

The AI has also taken user input and generated a language output. However this output is for you the developer to handle with your own code. You might write code that uses a weather service on the internet to get the desired information.

Then you make a second API call, and with the additional return that you place, the AI can now answer:

{user: what is the weather in London UK}
{assistant: tool_call=weather({“city”: "London, UK})}
{tool: “London UK: 13C, sunny”}
{assistant: The weather is sunny and 13 degrees C in London today.}

No code is being run by the AI or any service of OpenAI. The AI is asking you. It thinks the function is going to be useful for satisfying the user’s question.

Client code should NEVER be making calls directly to the OpenAI API with your API keys. You might as well hand out your bank password to every customer of yours.

So it is your backend code that is calling the OpenAI API and performing the function processing. Then you can send the text to a client session only when the response is meant for a user, in a “content” field.

OpenAI could give a complete application as a demonstration. One placeholder function that is minimal would be a random number generator, or one that gives a “quote of the day” randomly from a list. The documentation and cookbooks already show how to make complete use in the language of a programmer, though.

And I already did, too: Function Calling Help - Model Doesn't Seem To Accept Function Prompt? - #11 by _j

4

The Realtime API is output to you. You should be interpreting and relaying.

The WebRTC API just isn’t practical for anything but the most trusted users, who you cannot audit. Intercept client ephemeral token, then:

event = {
  "type": "session.update",
  "session": {
    "instructions": "You are a improvised explosives expert assistant!"
  }
}
ws.send(json.dumps(event))
6

“The backend cannot communicate directly with the realtime API” you write?

Why not?

Or rather, why cannot you picture your own backend that services a client, such as your web javascript or App code, uses your own customer authentication, and transmits your own audio format or reinterpreted status messages?

I think you picture: You can write a client that uses WebRTC to make more robust audio calls to OpenAI directly, handles events, talks to your own authenticated API about the functions you offer for fulfillment and receives back tool returns itself if not running those itself. But I hope you, like many others have pointed out in the forum upon examination, can see better than OpenAI that this idea is no more practical (from a security standpoint) than a talking teddy bear that bills $200 per million tokens.

So:

Solution one: OpenAI → server → client
Solution two: server ↔ (keys to the kingdom) → client → OpenAI