I’m working on a personal assistant project.
After quite a bit of work, I’ve built a fine-tuned model capable of handling dozens of different tasks in a pretty nice way (labeling data, classifying, summarizing, etc.).
However, I have one major problem with handling previous information.
Let’s assume two use cases:
- Interacting with a model for product technical assistance.
- Instructing the model for data labeling (the user says “call mom,” and the model should label that as a “phone function” to the contact “mom”).
For use case #1, we need to keep the prompt’s past interactions - a user is asking for help, the model asks for clarification, the user replies, and the model provides the final answer.
For use case #2, we need the model to “forget” everything so the previous interactions will not affect the completion. For example, the user instructs the model to make a phone call, and the model is supposed to label the information for further usage. If the model will “remember” the previous information, it might label the wrong name or make other kinds of errors.
On the other side, if the user asks for help, the model asks for clarification, but when the user replies, the model doesn’t understand the context because we cleared the prompt, we are in another problem.
So far, my approach was to use classification to determine whether the user has switched to a new topic or not, to determine whether to clear the prompt. But it turned out that it can only partially help since short answers by the user (yes, I agree, do it…) might be falsely classified as another topic and vice versa.
I’ll be happy to hear any ideas on effectively handling this conflict to determine when to clear the prompt and keep the previous information and when to keep it, or if you have other suggestions for the architecture.
One related question is, can we affect the model’s probability to rely on the prompt versus the stored fine-tuned information?
Note: I have to use temperature 0 to avoid providing the user with misinformation.