How do i Ground Realtime Voice to only answer from data returned by tools

API
1

How can i restrict the real-time voice model to only answer from the data returned by tools. if there is no data returned it should say it does not have information to answer. I want it to strictly follow the rules.
I am using gpt-4o-realtime-preview-2024-12-17

2

Hey! Just to clarify — have you already set up the part where your tool returns results (like from a vector DB or API)? Or are you also asking how to do that too?

Either way, here’s how I got the model to only respond based on tool output, and say “I don’t know” if the tool didn’t return anything:

  1. Define a tool in your session config:
tools: [{
  type: "function",
  name: "fetch_answer_from_data",
  description: "Look up information based on user input and return a response",
  parameters: {
    type: "object",
    properties: {
      query: { type: "string" }
    }
  }
}]
  1. Add strict instructions to the session
    This guides the model to rely only on the tool:
instructions: "Only respond using the output returned by tools. Do not use general knowledge or assumptions. If the tool returns nothing, say you don’t have enough information."
  1. In the tool handler, control the output
    When your backend handles the tool call:
const results = await vectorSearch(query);

const output = results.length
  ? `Here’s what I found: ${results[0].text}`
  : "Sorry, I couldn’t find any relevant information.";

return {
  output
};

  1. Send that back as a function_call_output
    Just send the final string.

  2. Trigger the response to be spoken:

{
  type: "response.create",
  response: {
    modalities: ["text", "audio"],
    conversation: "auto"
  }
}

This setup ensured the model only responded with what the tool returned — and nothing else if no data was available. Hope that helps clarify the approach.

3

This is what i tried and does not follow it.
export async function UpdateTools() {
try {
console.log(“UpdateTools function executed”);
var event = {
“type”: “session.update”,
“session”: {
instructions: “Instructions: Only respond using the output returned by tools. Do not use general knowledge or assumption. If the tools returns nothing, say you don’t have information. If user ask to stop, immediately stop.”,
“tools”: [
{
“type”: “function”,
“name”: “generate_horoscope”,
“description”: “Give today’s horoscope for an astrological sign.”,
“parameters”: {
“type”: “object”,
“properties”: {
“sign”: {
“type”: “string”,
“description”: “The sign for the horoscope.”,
“enum”: [
“Aries”,
“Taurus”,
“Gemini”,
“Cancer”,
“Leo”,
“Virgo”,
“Libra”,
“Scorpio”,
“Sagittarius”,
“Capricorn”,
“Aquarius”,
“Pisces”
]
}
},
“required”: [“sign”]
}
},

And if i ask what is gold, it answers in detail and i don’t have tool that provides that information.