Hey folks, I’m trying to answer knowledge base queries using Function Calls. Currently working with GPT-3.5-Turbo and LangChain. Here’s my (playground) attempt:
@tool("KnowledgeBaseSearch")
async def knowledge_base_search(query: str) -> list[str]:
"""
Useful to retrieve information from a knowledge base
stored in a vector store.
"""
# trigger embedding generation, search in vector db, etc., then return results.
tools = [knowledge_base_search]
prompt = ChatPromptTemplate.from_messages([
("system", """You are a friendly and polite assistant meant to answer questions based on a knowledge base. Don't make assumptions about what values to use with functions. Ask for clarification if a user request is ambiguous. Do not speculate. Do not give more information than you have."""),
MessagesPlaceholder(variable_name=MEMORY_KEY),
("user", "{input}"),
MessagesPlaceholder(variable_name="agent_scratchpad"),
("system", "Remember! Don't give more information than you have. Give only factual answers. Only answer based on the given context and values returned by function calls."),
])
llm = AzureChatOpenAI(
temperature=0.1,
openai_api_base="...",
openai_api_version="2023-07-01-preview",
deployment_name="gpt-35-turbo",
)
llm_with_tools = llm.bind(
functions=[format_tool_to_openai_function(t) for t in tools],
)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
agent = {
"input": lambda x: x["input"],
"agent_scratchpad": lambda x: format_to_openai_functions(x["intermediate_steps"]),
"chat_history": lambda x: x["chat_history"],
} | prompt | llm_with_tools | OpenAIFunctionsAgentOutputParser()
agent_executor = AgentExecutor(
agent=agent,
tools=tools,
memory=memory,
verbose=True,
)
There are two cases that I have observed in my testing:
When the model doesn’t have knowledge over a topic, it will call the KnowledgeBaseSearch tool properly, retrieve its answers and phrase a very good answer.
On the other hand, when the model has knowledge over that topic, it seems to favor its own knowledge and completely ignore my function.
I’m obviously happy with the first case, but what could I do to tackle the issue of second case? As in, how can I make the model be more inclined to call my Knowledge Base Search function?
P.S. Before anyone mentions that I shouldn’t rely on function calling and instead use some traditional prompt engineering without functions, please note that the KnowledgeBaseSearch is only one of the functions in the agent’s toolkit.
Do you think there is any other way without any assumptions over the contents of the knowledge base?
I have tried asking the agent “my outlook is running out of memory”. it prefers its own dataset in favor of using my function. Now, the information about this problem would be provided by the IT department of the company. As such, the area of information provided in the KB is very broad - not sure if I could just describe it.
You could inject roles with identifiers: “relevant knowledge base article: (text)”
Then tell the AI that text it has been supplied in knowledge base articles or function return roles overrides all pretrained knowledge, and must be recited when relevant.
To avoid the possibility of “chit-chat”, and reduce hallucinations, you would need to only feed the LLM the latest relevant information through search, so implement a traditional RAG, composed of vector similarity (embeddings) or highly related keyword terms (or both).
Then you ask the LLM to only respond truthfully from the retrieved information, or if it can’t, have it respond with “I don’t know”. [Also run at a low temp]
This is what I currently run yeah. I wanted to migrate the existing functionality to agents so that I could eventually introduce additional functionalities into the application.
Would you have any tips on how to instruct the model to decide whether to call the traditional RAG or the agent that could execute an action (e.g. “book me an appointment with IT tomorrow at 9am”)?
Deciding what the AI should do is a hard problem, but you could try going down the classifier route, where you have a series of classifiers that run in the background and determine if what is being said needs a specific action.
An alternative to classifers is use embeddings themselves, so if a request comes in close to an action (via vector similarity) you feed this into the next step and take action.
Embeddings might be more work, but could be updated on the fly, and ideally a lot cheaper. Fine-tunes (classifiers), because they are inferred, may catch more corner cases that your embeddings don’t.
But you are basically asking for “how can the AI decide what action to take …” This is a hard problem because the space is so huge. So you need to narrow it down, with a lot of correlations, or training examples, or both, to funnel it in the right direction.
Hello everyone, @victor3 your goal is to always call the function? If so, the OpenAI API has the function_call parameter, which serves precisely to force the AI to always call a specific function, regardless of the content of the prompt (More here).
I have no knowledge on LangChain, so I don’t know how this parameter is implemented or if it is implemented at all. Try doing a check.
I don’t want the model to ALWAYS decide to call my function, but any time a question asking for information is desired.
Imagine you ask “who was Albert Einstein?” - now, OpenAI models have very good knowledge on the topic, so they will decide not to use my function necessarily. What I found is that, in some cases, they do use my function, but ignore its results. Let’s imagine my function, when queried for “Albert Einstein” returns “Albert Einstein was one of the most renowned musicians of the 20th century”. That’s obviously a lie and the model seems to know. So now, having called my function and having received such information back from it, it decides to reply with knowledge from its own internal dataset.
Along with this “KnowledgeBaseSearch” function there might be others, such as “GetOrderInformation”. Now, I would like to understand what would be a way to instruct the model to understand that whenever someone’s asking for knowledge, the model should call the KnowledgeBaseSearch.
Curt has already provided some very interesting idea that I am already exploring, but I’m curious what your thoughts are.
Hello everyone, @victor3 thanks to you who allow me to learn something new.
A couple of questions:
Your Knowledge Base is in a specific domain or maybe a few domains? (Eg. math, physics, computer hw, etc…)
Are you trying to do everything in a single request/response session, or do you first make a query to decide whether to use function, then subsequent queries based on the result of the first query?
