Realtime API with Knowledge Base

API
, ,
1

I am working on the code of realtime API twilio integration. I want to call here an external api or use open ai knowledge base / vector store or db with it.
Like user ask for some query and that will respond from its knowledge base
How can I implement that? step by step.

1 Like
2

Add some functions in your realtime session configuration or per-request. When the model calls a function, have this function contain some query-like info from the user. You can then use this information to call any API’s or queries to whatever you like. If the latency allows, you can take the data acquired by this retrieval and send the response back to the model by specifying that this is a response to the function call

2 Likes
3

and how do I use functions with realtime API
Like this is normal code
// Control initial session with OpenAI
const initializeSession = () => {
const sessionUpdate = {
type: “session.update”,
session: {
turn_detection: { type: “server_vad” },
input_audio_format: “g711_ulaw”,
output_audio_format: “g711_ulaw”,
voice: VOICE,
instructions: SYSTEM_MESSAGE,
modalities: [“text”, “audio”],
temperature: 0.8,
},
};

  console.log("Sending session update:", JSON.stringify(sessionUpdate));
  openAiWs.send(JSON.stringify(sessionUpdate));

How to use openAI functions

4

All the necessary information is provided in the API reference