It produces responses with incorrect numbers, in my case, these are prices

Prompting
1

Hello,

I’m working on a bot that should answer questions. I’ve created 100 variations of questions and answers. An example is:

“{”“messages”“: [{”“role”“: ““system””, ““content””: ““Anna is a real chatbot who is also friendly and a little sarcastic.””}, {”“role”“: ““user””, ““content””: ““What is the price for adults entering the rope park ?\n\n###\n\n””}, {“role”: “assistant”, “content”: ““The admission to the rope park a alone costs €26.00 (twenty-six euros) for adults. If you’d like to include the cable car ride, it is €35.00 (thirty-five euros). For a more extensive adventure, there’s the option with admission to the rope park, a mountain ascent, and a mountaincart descent for €41.00 (forty-one euros).\n””}]}”

Unfortunately, it often provides incorrect prices and sometimes invents four more of its own variations

2

Your last message dict is incorrect. You set the role as your message and miss “content”.

2 Likes
3

The main issue with this approach is that you expect fine-tuning will change the model to answer pricing information correctly.
As you are experiencing this is not the way to teach a model new knowledge. You should use retrieval augmented generation instead.
Also, imagine the consequences of the inevitable price change next season.

1 Like
4

Thank you for the comments, and I apologize; in this example, I made a mistake (already corrected). There is no such error in the JSON file.

The trained model tends to respond similarly to the example, but it frequently makes mistakes when dealing with numerical values, especially with prices.

5

The number of "s at the last tuple seems a bit off, also, for the problem with numbers may be at the prompt layer, so try to ask for the model to review it focusing on the input values, maybe try this review request with a lower temperature, hope this helps :blush:

1 Like
6

You are very likely to waste time and money trying to reach your goal using fine-tuning.

Here are some relevant topics to read up on.
Of course they don’t match your case exactly but the general direction will be helpful for you.

Edit: The explanation is that every time you want to get the price for the waterslide ride the model will guess the most likely response. But what you want is the definite correct price. Every single time.

2 Likes
7

Hi!

One option, not sure is the best one, for this case would be the following:

Create a database of prices, such as a JSON or CSV file, especially if the variety of prices isn’t too extensive. Here’s an example for the JSON file:

{
  "price_for_adults": {
    "standard": 26.00,
    "price_with_cable_car": 35.00,
    "full_adventure_price": 41.00
  }
  //  "price_for_kids", "price_for_families",
}

Then create a new agent modifying your prompt system. Your system prompt should guide the bot to understand which specific price is being inquired about by the user and to return a JSON object (or a string) with that price information. An example system prompt could be something on the line of:

{
  "role": "system",
  "content": "Your task is to interpret the user's query about pricing. Based on the query, return a JSON object specifying the requested price type. The JSON structure should be: { 'price': 'select one from [price_for_adults, price_for_kids, price_for_families] }"
}

Following this, the user’s question comes in, and the bot’s response should be based on the selected price type from your JSON or CSV file. You might consider using a secondary model to generate more natural-sounding responses that incorporate the selected pricing information.

This above is an example, you probably need to change it a bit and request the price only when needed.

2 Likes