Hey everyone, I’m working with the new responses API and trying out the out-of-the-box file_search tool. Here’s what I’ve done so far:
Set up a vector store and added the file_search tool schema
File search is working, I can see that it’s retrieving relevant knowledge correctly
Overall, technically things are functional
Where I’m running into trouble is in how the AI is using (or rather, overusing) the tool in practice.
In my use case, users aren’t uploading any files themselves. There’s a background job that manages file ingestion and indexing centrally, kind of like a shared knowledge base. But when the AI responds, it keeps referencing “files uploaded by the user,” which is both confusing and inaccurate for my setup.
Also, I only want the AI to use the file_search tool in specific scenarios (which I’ve outlined in the system prompt). But what I’m seeing is it reaching for the tool way too often, sometimes even proactively saying things like “It looks like some files have been uploaded. Would you like me to look for something specific in them?” — which doesn’t make sense in my context and makes the experience feel off.
I tried guiding the behavior via the system prompt with stuff like:
“The file_search tool allows you to search and retrieve relevant existing knowledge that has been submitted by others in the organization. These are not uploaded by the user, they are maintained centrally and provided automatically for reference and inspiration. Only use the file_search tool if the user explicitly asks to explore…”
…but I’ve had little success. It seems like either there’s some internal system prompt baked in when the file_search tool is enabled, or the model has been fine-tuned to assume a certain behavior that’s overriding my instructions.
Curious if anyone else has run into similar issues or has tips for better controlling tool use and messaging? Would love to hear how others are handling this.
In our use case, we typically WANT the model to use the file_search on virtually every generation, so we haven’t run into this problem. We did, however, run into the opposite problem – that sometimes it would not use the file_search tool when we wanted it to. We solved that problem with a very simple instruction to “Use the file_search tool” (more or less). My reaction to the instructions that you shared is that it is too complicated. First, ask yourself how the model should know when to use the tool, then ask it to do exactly that. Suppose you just said, “Use the file_search tool if the user is asking to…” – and figure out how to finish that simple sentence.
Again, I know how much time we spent on instructions before we got them right, so I know it is not easy. But our experience is that short definitive instructions are the path to success.
IMHO, Instructions are key. Did you find that saying Use the file_search tool if the user is asking to… or something like “Only use the file_search_tool. Do not use any other source” works? I work for a medical research hospital and if any researcher loaded their PDF’s or a group of PubMed Articles but got an answer that was external and not what they were thinking then the app would never be used. I need privacy on the personal uploaded documents and 100% accuracy to the public files inputted.
“100% accuracy”. Good luck with that. The nature of AI is to generalize. If you can’t afford mistakes, you may be using the wrong technology. Even humans are going to make mistakes trying to interpret documents.