Hi! I am using the Assistants API and have created a vector store which contains reference material my Assistant can draw from to answer user queries. If the files in the vector store contain information relevant to the query, it should use them, but otherwise, it should answer from general knowledge.
My problem: when the Assistant does not find an answer in the files, it tells the user that the files didn’t contain the information, which I don’t want it to do.
Examples:
The files provided do not contain X
It seems that the files you uploaded contain detailed instructions on X, but not Y
The documents do not specifically mention X
Additional context: The end user does not know or care that there are files in the vector store, they just want the answer to their query.
Any suggestions? I have tried adding various instructions to the prompt, e.g. 'You have access to the attached files which you can use to answer user queries. If there is no relevant information in the files, just answer from your general knowledge.; but this doesn’t help/
So, what exactly is the information being retrieved? As in, what kind of assistant are you trying to generate?
Could you also share your system / developer prompt? I think I know what’s going on, but I want to confirm. There’s something in the way that instruction prompt is being set up to where the model is confusing the role its meant to be playing.
Hey, thanks for your response! Unfortunately I’m unable to share specific details on the assistant or system prompt, but it is a single conversational agent designed to engage in conversation and answer user queries on a particular topic, much like a customer service agent. The system prompt contains an overview of the assistant’s role, guidelines, and constraints, and does not mention anything about the files uploaded to the vector store.
I suspect that the ultimate solution is a multi-agent approach with a dedicated retrieval agent, but I’m wondering if there is a workaround stop these types of responses in the short term. Any hypothesis you can share would be much appreciated!
The instructions they give to the AI model about the file search tool says directly within that “the user uploaded files”.
You can spend a lot of effort at trying to counter the file search tool text which you cannot fix by giving a bunch of instructions.
Or you can spend the effort to make a RAG solution portable to chat completions instead of Assistants, or portable to any AI provider’s platform who is more responsive to such concerns.
Hand off the response to a smaller faster model like 4o mini with a simple chatcompletion api call and have it sanitize it by excluding any mention of files. With a good prompt there you will never see any mention of files.
I can tell you that we solved this problem with a careful design of our instructions. The best thing you can do, in my experience, is to be very clear about what you want and do not want. Without seeing your instructions, it’s hard to say what might be improved, but I encourage you to experiment with very simple instructions first that establish clear simple goals. As far as the RAG content, just instruct the LLM to use the file_search tool to look for relevant content and use it if it is found. We found that as you make the instructions more complicated, it becomes more likely that it locks onto certain specifics within the instructions rather than keeping the “big picture”. BTW, we use gpt-4o-mini, so we’re not using a big fancy model.