Hallucination in retrieval augmented chatbot (RAG)

[SomebodySysop]

@shamy

When I pass conversation history, I am not passing the context it used to generate the answers - only the queries and responses. I am doubting that the model looks at it and thinks it hallucinated the past answers and hence starts to hallucinate. Do you think that is a possibility?

I maintain the chat history, but I don’t send it to the model (with the question) – instead, I use it to create a standalone question which I send to the model along with the context documents. I’ve not had any problems (ever) with hallucination in my RAG implementation. When the answer isn’t in the retrieved documents, the model (whether gpt-3.5, gpt-3.5-turbo or gpt-4) always responds as it it is instructed by the system prompt (which is also always sent with the standalone question).

How to construct the prompt for a standalone question?

This is how I’m using it. It appears to be working on a variety of documents. And does really, really well in maintaining the conversation context, as you can see below: [SolrAI: How the chat completion process with standalone question maintains conversational context]