Hi, I’m trying to build a dummy chatbot with 1 use case. I’ve followed the documentation. I’ve uploaded the fine-tunned but once I test it in the playground I receive wrong answers.
This is the json line used:
{"prompt":"Summary: the customer wants to add a bag in his booking.\n\nSpecific information:a record locator is required. The record locator is an alphanumeric of 6 characters.\n\n###\n\nCustomer: hello, I want to add a bag in my booking\nAgent:", "completion":" Do you have a reservation number? Provide the record locator please.\n"}
{"prompt":"Summary: the customer wants to add a bag and has provided the record locator.\n\nSpecific information: a bag size is required\n\n###\n\nCustomer: hello, I want to add a bag in my booking\nAgent: Do you have a reservation number? Provide the record locator please.\nCustomer: A45TGH\nAgent:", "completion":" Could you provide the size of the bag to add?\n"}
{"prompt":"Summary: the customer wants to add a bag and has provided all the required information.\n\nSpecific information: the bag will be added soon\n\n###\n\nCustomer: hello, I want to add a bag in my booking\nAgent: Do you have a reservation number? Provide the record locator please.\nCustomer: A45TGH\nAgent: Could you provide the size of the bag to add?\nCustomer: A45TGH\nAgent:", "completion":" Your bag would be added soon.\n"}```
And this is the result:
This is a chatbot:
Customer: hello, I want to add a bag in my booking\n\n###\n\n
Agent: Hello, I am the agent. How can I help you?
Customer:
I guess I need more examples but I thought that would be enough to receive correct answers for the main scenario which is "to add a bag in my booking".
Thanks in advance!
It doesn’t look like you’re following the same format. In your prompt example you open with “Summary:” but that doesn’t appear to be present in your example completion. Following the same exact format is critical.
So why the summary and the specific information is used for? I thought it was used to give context of the situation and the prompt.
At the end the customer will do a simple question and the idea is that the chatbot is able to provide an answer based on what it is specified in the fine-tuned model.
With fine-tuning you must train it with examples that are exactly the same format as what you want the production examples to be. Unless you plan to include “Summary” and “Specific information” from your production input then you need to remove it from the training set.
You can see that the only framing I add is “QUESTIONS:” at the end of the prompt. This is so the finetuned model learns that is where the conversation/context ends and the questions begin.
{"prompt":"Summary: <summary of the interaction so far>\n\nSpecific information:<for example order details in natural language>\n\n###\n\nCustomer: <message1>\nAgent: <response1>\nCustomer: <message2>\nAgent:", "completion":" <response2>\n"}
{"prompt":"Summary: <summary of the interaction so far>\n\nSpecific information:<for example order details in natural language>\n\n###\n\nCustomer: <message1>\nAgent: <response1>\nCustomer: <message2>\nAgent: <response2>\nCustomer: <message3>\nAgent:", "completion":" <response3>\n"}
I’ve tried to use the “recommended” prompt for finetuning, too, following “best practices” from the OpenAI docs using the “###” delimiter and something that I call a “increasing completion convo” (msg1 > msg1+2 > msg1+2+3 > …). Basically everything mentioned in the docs.
This generated about 90 prompts. Even though you should do a few hundred, it performed very very poorly, to say the least. It kept responses extremely short and was almost sarcastic, even though the temperature was topped at 1. Changing it didn’t affect much.
I wasted almost $2 on this, whatever… Learning…
Don’t do it this way.
As @daveshapautomator mentioned in one of his videos, “###” is basically meaningless to the model and using the initial explanation copied into every prompt is not needed.
I did another experiment using only 20 examples, removed the initial explanation and used only the conversation.
The prompt being the whole conversation but the last and the completion being the last.
Basically this (also from dave) : InformationCompanionChatbot/eve.jsonl at main · daveshap/InformationCompanionChatbot (github.com)
I then put the initial explanation inside the Playground (also something that I saw in dave’s videos)
In this go I did just 1 epoch (default is 4), and it performed better. It still didn’t use my opening phrase that I specified 20(!) times, but at least it responded better.
What this lead me to think is that it might not be wise to use opening phrases inside the fine-tuning, at least not with such small datasets.
Maybe it doesn’t “relay” abstract thoughts or exact phrases well from fine-tuning with small datasets… I still don’t know…