I am currently experimenting with creating an assistant that can answer questions about our software. We have a pretty large amount of Q&A in an Excel file we have supplied to the Assistant. In this file, every question a customer has asked is a row, while the answer (or answers) given to this question are included as separate rows. Sometimes a conversation occurs between the client and us, resulting in multiple back-and-forward messages until we can finally give a definitive answer and close the ticket. That’s why one question can have multiple rows.
I have asked the Assistant about how I should supply this data, and it said to use multiple rows in the Excel-file and make sure they have a common denominator, so the Assistant knows that the rows are related to each other. I have done this.
Along with this, I have also created an extra file with everything the Assistant needs to know about our software to give good answers to user queries.
The Assistant is configured to first check the second file with basic information about the software for an answer. If it can’t find the answer there, it should consult the Q&A-file for an answer.
The results we are getting with this approach are very mixed. On the one hand it does reference the knowledge in the files at times, and at other times it seems to miss the mark completely and fabricate an answer, even though the right answer is in the first file or the Q&A-file.
Therefore I have also experimented with fine tuning today, by supplying a list of manuals to the software. After processing, I notice in the Playground that the quality of the responses has increased somewhat, but it seems to miss the mark far more often, saying stuff that might be true for other software applications, but not ours. Sometimes we also receive a partially incorrect answer where step 1 and 2 are right and step 3 is wrong.
Because of this, I am now wondering what path to pursue? Should I focus on iterating over my fine-tuning jobs to get the results I need, or should I focus on working with the assistant to get a qualitative supportdesk-assistant after some time.
Can anyone share their view on this or share their experiences with this type of use case?
