What’s the number one challenge that you face when you create and roll out chatbots that interact with users using API?
I will start with mine.
In my case, one practical challenge is evaluating the prompt upon making on-the-fly changes.
While testing my chatbot, If I encounter some hallucinations, like providing information that doesn’t exist in the system prompt (Eg, Support email IDs, links, etc), I try to modify the prompt so that it can provide the information in future conversations. Or maybe something else that changes the behaviour.
But, it becomes difficult to evaluate the impact of such on-the-fly changes.