Knowledge Base in realtime api

API
, ,
1

Hello everyone,

I know this question has been asked before, but I couldn’t find a helpful answer. I’m currently working on building a voice agent using Twilio and OpenAI’s real-time API.

Now, I want to integrate a knowledge base into the agent, such as various types of documents. How can I achieve this?

Any guidance would be greatly appreciated!

2

You are most likely looking for this:

dataChannel.addEventListener('message', async (ev) => {
    const msg = JSON.parse(ev.data);
    // Handle function calls
    if (msg.type === 'response.function_call_arguments.done') {
        const fn = fns[msg.name];
        if (fn !== undefined) {
            console.log(`Calling local function ${msg.name} with ${msg.arguments}`);
            const args = JSON.parse(msg.arguments);
            const result = await fn(args);
            console.log('result', result);
            // Let OpenAI know that the function has been called and share its output
            const event = {
                type: 'conversation.item.create',
                item: {
                    type: 'function_call_output',
                    call_id: msg.call_id, // call_id from the function_call message
                    output: JSON.stringify(result), // result of the function  <<<<< here you can add your RAG and let it give extra information I guess
                },
            };
            dataChannel.send(JSON.stringify(event));
        }
    }
});

watch this please:

I mean I didn’t try so it is a theory - you have a function call - and the function call can also call an OpenAI API and when it can do that you can also add a RAG…

In PHP that could look as beautiful as this:

$chatResponse = $this->chatCompletionServiceFactory
   ->create('openai') 
   ->generateResponse(
       $this->chatMemoryServiceFactory
          ->create('graph') // or use a vectorDB RAG or a SQL RAG
          ->getChatHistory(
             ChatMessageEntry()
                 ->addRole('user')
                 ->addMessage($prompt)
             )
   );
2 Likes
3

Hi,
So there are 2 main methods in which you can integrate a knowledge base into your agent, you can go for a RAG based approach where you can look into: Using Redis for real-time RAG goes beyond a Vector Database - Redis, the other option is utilizing Knowledge Graphs which would provide you slower service but can change depending on your use case: Knowledge Graphs & LLMs: Real-Time Graph Analytics

I think you can also look into a hybrid approach depending on your requirements and circumstances.

4

Have you been working on a GraphRAG yet?

The image shows a user interface for a Systemic Functional Linguistics Graph with a text input field labeled "gib" and a section titled "Ideational Metafunction Assignment." (Captioned by AI)

Working on that kind of stuff at the moment… like adhoc analysis of chat data and creating a graph to simulate short term memory

Want to determine what of the chat can be used to finetune a model… want to use the ChatGPT export on that to have a RAG with everything I was talking to ChatGPT with and then doing semantical clustering…

1 Like