How to incorporate user feedback into my RAG pipeline

API
1

I have a simple RAG pipeline, but now say the user is not satisfied with the response, how can i incorporate this feedback to improve my RAG in a continuous manner? Thanks

2

Well, embedding search is essentially just a hash map.

You can just disconnect and reconnect nodes as you see fit. I suppose a big question is, can you trust your users to give you reliable feedback?

3

Sorry can you explain further what you meant? i dont get ir

4

I think there are no methods that are generally considered best practices, I don’t think too many try to do what you are trying to do. Some options come to mind

(1) Generate again with temperature > 0

(2) Ask LLM to do self-reflection on the (2.1.) original topic (2.2) based on the unsatisfactory answer and then do RAG search again by including the self-reflection text

(3) Save which chunks were used to generate the response. If user clicks thumbs down, then take chunks that are the next closest. I think this is what @Diet means

For general improvement of RAG results;
(4) Use other types of search or filters than embeddings based, even keywords may work in big databases.

(5) Do RAG differently (use only keywords vs. use full conversation etc.)

(6) improve chunking (overlapping chunks, chunk on paragraph breaks or page breaks, or chunks based on topic change etc.

How to exactly do it depends on are you writing most of the code yourself, or do you use some libary, like LangChain or Semantic Kernel

1 Like
5

Have you got the solution
Like , I am getting same Issue to implement the user feedback mechanism to improve the future RAG responses.