Hey there, I’m looking at a few options for optimizing an IT support RAG chatbot.
What I have implemented presently is a RAG system with the companies internal knowledge base. This feels already pretty well optimized to me using some custom HyDE and reranking techniques to consistently retrieve the ideal document chunks. Im also using a chain of thought response with a long answer followed by a summarized answer, the users just see the summarized answer.
It still fails on tricky troubleshooting tasks around dashboards and data discrepancies, basically throwing out all the possible troubleshooting steps at the user rather than knowing which ones are most likely to solve the user’s problem. To combat this I tried augmenting the company knowledge base by generating knowledge articles from past support tickets, I would say this made a very small difference but there is still room for improvement. I’m trying to decide which of the following options to explore to improve the chatbot further:
- Finetune a model using the ai generated support ticket knowledge articles where user messages are the problem description and the assistant messages are the troubleshooting steps.
- Explore an agentic approach where if needed the model can query our snowflake DB to pull in a data sample related to the dashboard the user is troubleshooting.
- Something else? any other ideas?