I’m building a support bot that depends heavily on RAG. What I’m finding is that it works great when the LLM uses its RAG function and retrieves external data from our knowledge base. But sometimes it just doesn’t use the RAG function when it should. In those cases it gives generic answers that aren’t very useful, which makes sense. Is there any way to make it use RAG more often? I would like it using the RAG function for every question other than simple greetings that obviously don’t require external data to answer properly.
1 Like
Hi @rybo
You can specify in the system message when to use the function <function name> based on the context of the message.
E.g. “Use function retrieval() only when the user talks about <topic name>”
You can also add an “otherwise” clause to tell the model what it should do if user talks about something else.
By “otherwise” clause, do you mean as natural language in the system prompt?
Currently in the system prompt I have
Always use the local_forum_search function to find the answer on this forum. This forum is your best source of information. Do not look anywhere else. Do not give generic answers because you can get better answers by searching this forum.
It seems to use the function most of the time. It almost always does on the first request when I start a conversation. Often it will call the function in the first request but when I follow up with a second request in the conversation it won’t call the function the second time.
I’ll try wording it more like your example.
I also just noticed in the API docs there is a tool_choice field that might force it to call the function. That might be what I’m looking for!
Yes I mean it as a NL prompt.
Good find on the tool_choice param. If set to the name of the function, it will force the model to call that function.
1 Like
Building on @sps 's response, I believe this boils down to your instructions/prompt. I’d suggest prompting your thread/assistant (assuming you’re using the assistant’s API here) to follow some procedure with each request. Something like;
"For each X input provided by the user, follow this procedure:
- Lookup KB information with
funcion() - Assess if the information returned by
function()is relevant to the user’s query - Summarize the KB information to user in XYZ format. "
I like to use the COSTAR method when creating instructions for my assistants. I’ve had really good luck with that framework: Unleashing the Power of COSTAR: A Journey into Structured GPT Prompting | by Trail Techie | Jan, 2024 | Medium
1 Like
I updated the system prompt like this. It’s a good idea but it still doesn’t seem to make the bot use the RAG function most of the time. 
It looks like we’re going to have to use the tool_choice option to force it.
Do you mind sharing a snippet of what you landed on? Would most likely be helpful some day in the future to understand what works well! Thanks!
Sure, I realized that we can’t use the tool_choice option because there are situations where we need other functions to be called. Using tool_choice would always force the LLM to search the knowledge base and never allow it to call other functions.
So I’m going with temperature = 0 and a rather large system prompt to try to compensate for the LLM not searching our knowledge base as often as I would like.