Fine-tuned model not giving expected results

API
,
1

Hello,
I’m new to fine-tune a model using GPT-3.5-turbo.

What I want to build

I want to build a tool that generates name ideas for an app based on a given context
(for example I provide “Fitness app” and in response I get list of good brand names for the app).

Why I want to fine-tune my own model and not just use chatGPT api ?

when it comes to brand names generation chatgpt don’t give prefessional, high quality names, it gives you basic ones.
so I said maybee I can train it using the top names on the market to get inspiration on how should the names be.

what I did

I scraped top 30 top apps and got their names on multiple categories (fitness, video editors, etc…)
the training dataset looks similar to this:

{
  "messages": [
    {"role": "system", "content": "You are a smart brand name generator. I will provide you with the name of my app and you will generate me new unique names that are not in your knowledge. Provide results in JSON => [{name: DATA, explanation: DATA},...]"},
    {"role": "user", "content": "photo editor app"},
    {"role": "assistant", "content": "[
      {"name": "Photoshop", "explanation": "photoshop is a combination of photo and shop and it means ...."},
      {"name": "Lightroom", "explanation": "explanation here ......"},
      {"name": "Pixlr", "explanation": "explanation here ......"},
      {"name": "Canva", "explanation": "explanation here ......"},
      {"name": "Pixart", "explanation": "explanation here ......"},
      etc...
    ]"}
  ]
}
......etc

Using the model (Input)

now I just used the model like this:

{
  "messages": [
    {"role": "system", "content": "You are a smart brand name generator. I will provide you with the name of my app and you will generate me EXACTLY 50 new unique names that are not in your knowledge. Provide results in JSON => [{name: DATA, explanation: DATA},...]"},
    {"role": "user", "content": "photo editor app"},
  ]
}

Model response (Output)

[
  {"name": "Pixlr", "explanation": "explanation here ......"},
  {"name": "Canva", "explanation": "explanation here ......"},
  {"name": "Photify", "explanation": "explanation here ......"},
  {"name": "PhotoXPro", "explanation": "explanation here ......"},
  {"name": "Gimp", "explanation": "explanation here ......"},
  {"name": "Prisma", "explanation": "explanation here ......"},
  {"name": "Luminar", "explanation": "explanation here ......"},
  {"name": "PixelMator", "explanation": "explanation here ......"},
]

What’s the problem ?

there is 2 problems! The first related to number of generated names, as you can see in my example I included “generate me EXACTLY 50 new unique names”,
but the number of generated names is very random, sometimes it generates 5, sometimes 20, sometimes 60.

The 2nd problem is the generated names, they are names of actual apps that are already used, and I specifically include: “names that are not in your knowledge”
so that it won’t give me names that are used by other apps and instead generate new ones from scratch inspired by the ones on the model.

2

Hey Champ and welcome to the community forum!

Problem 1.:

The models aren’t good at counting, you may be able to tell the model to list them with numbers, like asking “generate a list of 50 new unique names” but it’s not going to be bulletproof. If you really want to generate exactly 50 names, then it’s probably better to make 50 separate API calls.

Problem 2.:

One of the purposes of fine tuning is to make the models responses align with your training data, so this is actually the expected behavior. My best advice would be to try an generate the names based on a description of the companies instead.

I hope that helps!

1 Like
3

Thank you for your fast response, I really appreciate it.

This is a very good solution, in fact this problem of the number of provided results is fixable in many ways, the easiest one is your solution, but I can twist it a bit and instead of calling the api 50 times I can call it one time and keep counting the results, if it’s lower than 50 then I call the api again… until I got the number that I’m looking for.

you mean something like this ?

{"role": "user", "content": "empowers users to enhance, manipulate, and create stunning visuals with a wide array of tools. Perfect for photographers, designers, and creative enthusiasts, it offers limitless possibilities for digital artistry."},
{"role": "assistant", "content": "Photoshop"}

{"role": "user", "content": "powerful open-source software for image editing and graphic design. With a user-friendly interface and a wide range of features"},
{"role": "assistant", "content": "Gimp"}

{"role": "user", "content": "premier photo editing and organization tool. Tailored for photographers, it offers powerful editing capabilities, precise color adjustments, and efficient cataloging. Streamline your post-processing workflow and achieve stunning results with ease."},
{"role": "assistant", "content": "LightRoom"}
4

Always happy to help!

Almost, but you should really try doing this without fine tuning, here’s an example I made, just for you :wink:

https://platform.openai.com/playground/p/aHlkecLSHhwjKiNacvbywBAf?model=gpt-3.5-turbo

1 Like