Chatbot with a fine-tunned model

Hi, I’m trying to build a dummy chatbot with 1 use case. I’ve followed the documentation. I’ve uploaded the fine-tunned but once I test it in the playground I receive wrong answers.
This is the json line used:

{"prompt":"Summary: the customer wants to add a bag in his booking.\n\nSpecific information:a record locator is required. The record locator is an alphanumeric of 6 characters.\n\n###\n\nCustomer: hello, I want to add a bag in my booking\nAgent:", "completion":" Do you have a reservation number? Provide the record locator please.\n"}
{"prompt":"Summary: the customer wants to add a bag and has provided the record locator.\n\nSpecific information: a bag size is required\n\n###\n\nCustomer: hello, I want to add a bag in my booking\nAgent: Do you have a reservation number? Provide the record locator please.\nCustomer: A45TGH\nAgent:", "completion":" Could you provide the size of the bag to add?\n"}
{"prompt":"Summary: the customer wants to add a bag and has provided all the required information.\n\nSpecific information: the bag will be added soon\n\n###\n\nCustomer: hello, I want to add a bag in my booking\nAgent: Do you have a reservation number? Provide the record locator please.\nCustomer: A45TGH\nAgent: Could you provide the size of the bag to add?\nCustomer: A45TGH\nAgent:", "completion":" Your bag would be added soon.\n"}```

And this is the result:

![image|561x193](upload://xfoeHIDpJBVmUtjJtNBkWq5ALQp.png)

This is a chatbot:
Customer: hello, I want to add a bag in my booking\n\n###\n\n
Agent: Hello, I am the agent. How can I help you?
Customer:

I guess I need more examples but I thought that would be enough to receive correct answers for the main scenario which is "to add a bag in my booking".

Thanks in advance!
1 Like

It doesn’t look like you’re following the same format. In your prompt example you open with “Summary:” but that doesn’t appear to be present in your example completion. Following the same exact format is critical.

1 Like

So why the summary and the specific information is used for? I thought it was used to give context of the situation and the prompt.

At the end the customer will do a simple question and the idea is that the chatbot is able to provide an answer based on what it is specified in the fine-tuned model.

With fine-tuning you must train it with examples that are exactly the same format as what you want the production examples to be. Unless you plan to include “Summary” and “Specific information” from your production input then you need to remove it from the training set.

Take a look at a question generating set I created here: NLCA_Question_Generator/questions.jsonl at main · daveshap/NLCA_Question_Generator · GitHub

You can see that the only framing I add is “QUESTIONS:” at the end of the prompt. This is so the finetuned model learns that is where the conversation/context ends and the questions begin.

1 Like

But this format is suggested in the documentation for [

Case study: Customer support chatbot

](OpenAI API):

{"prompt":"Summary: <summary of the interaction so far>\n\nSpecific information:<for example order details in natural language>\n\n###\n\nCustomer: <message1>\nAgent: <response1>\nCustomer: <message2>\nAgent:", "completion":" <response2>\n"}
{"prompt":"Summary: <summary of the interaction so far>\n\nSpecific information:<for example order details in natural language>\n\n###\n\nCustomer: <message1>\nAgent: <response1>\nCustomer: <message2>\nAgent: <response2>\nCustomer: <message3>\nAgent:", "completion":" <response3>\n"}