Chat GPT accuracy of answer and memory

API
, , ,
1

Hi, I’m creating a ChatBot based on school law information. But my Bot has 2 problems:

  1. It doesn’t know how to use memory and when I ask my previous question it returns his previous answer
  2. Doesn’t give me the right answers every time.
    I’m using ConversationalRetrievalChain and Chroma for this, can you tell me where I’m going wrong, I’m still new to this.

This is how my prompt template looks:


from langchain.prompts import PromptTemplate
prompt_template1 = """Answer the question as accurately as possible from the context below
{context}
Question: {question}
Chat History:
{chat_history}
Answer in Polish:"""
PROMPT1 = PromptTemplate(template=prompt_template1, input_variables=["context", "question","chat_history"])
chain_type_kwargs = {"prompt": PROMPT1}

And ConversationalRetrievalChain:

memory = ConversationBufferMemory(memory_key='chat_history', return_messages=True, output_key='answer')

qa_memory = ConversationalRetrievalChain.from_llm(
    llm=llm,
    memory=memory,
    chain_type='stuff',
    retriever=retriever,
    combine_docs_chain_kwargs=chain_type_kwargs
)

Is it well written or should I change something? I’m wondering how I can give more hints to the model because Combine_docs_chain_kwargs=chain_type_kwargs only allows you to specify one key pair value.

2

Welcome to the developer forum!

For history to work you need to feed the previous prompts and replies back into the next prompt, I don’t use Langchain so I’m not sure how that is handled there.

For more accurate retrieval of information you should try overlapping your embeddings and possibly look at testing performance to evaluate various overlapping amounts, try with 50-60% overlap to start with. Again, I don’t know how this is handled with Langchain. Personally I have not used it as the ease of use it offers is quickly outweighed by not knowing exactly what is going on under the hood without looking through mountains of other peoples, often undocumented, code.

1 Like
3

When using Langchain, you don’t need to feed previous prompts to buffer memory.
I have implemented memory to load_qa_chain.
I suggest to add input_key parameter to memory definition.
Like
memory = ConversationBufferMemory(memory_key=‘chat_history’, return_messages=True, output_key=‘answer’, input_key=‘question’)

1 Like
4

Okay, understood, but how i this way I can refer to question i asked before ?
Something like:
Q1: What article 400 says?
Q2: What was my question before ?

5

You’ll probably have better luck asking on the LangChain forums since the issue is specific to their implementation of conversation memory.

1 Like