Fine-tuned bot answers way too short through API

[rektalizerofficial]

Hello, I am using fine-tuned davinci model for learning purposes. The problem I faced is that while playground version of my model answering with long and close to dataset type of answers, the telegram version that works through API shorten it’s answers a lot. For example playground version answered with about 200 words, while telegram one keep it close to 20-30. I ensured they both have same settings. The only difference might be that telegram version takes previous dialogue as a prompt, however it’s still an issue even if it’s fully restarted without any old dialogues in prompt.

[sps]

Welcome to the community.

It’ll be very helpful If you were to share the API request.

[ruby_coder]

Hi @rektalizerofficial

As @sps kindly requested we need you completion params to accurately assist; especially the temperature (s).

In addition please share your fine-tune params including the n_epochs param and a few example lines from your JSONL training file.

With that info we can help you for sure! Otherwise it is just “wild guessing”.

[rektalizerofficial]

Of course. Here is params:
n_epochs is not specified and should be default, which I believe is 4?
temp = 0.8
max tokens = 500 (was 256, but it didn’t change much)
stop = END (which is a stop sequence for my dataset)
frequency penalty = 0.5
presence penalty = 0.5

With same params the playground version still able to answer with long and well-rounded text. This one usually go for short and ignorant answers. Here is my API request:

def openAI(prompt):
    response = requests.post(
        'https://api.openai.com/v1/completions',
        headers={'Authorization': f'Bearer {API_KEY}'},
        json={'model': MODEL, 'prompt': prompt, 'stop': STOP_SYMBOLS, 'temperature': 0.8, 'max_tokens': 500, 'frequency_penalty': 0.5, "presence_penalty": 0.5}
    )
    print(response)

    result = response.json()
    print(result)
    final_result = ''.join(choice['text'] for choice in result['choices'])
    return final_result
...
 if '@RektGPT_BOT' in result['message']['text']:
                        prompt = result['message']['text'].replace("@RektGPT_BOT", "")
                        print(prompt)
                        # Calling OpenAI API
                        bot_response = openAI(f"{prompt}")
                        # Sending back response to telegram group
                        print(telegram_bot_sendtext(bot_response, chat_id, msg_id))

[ruby_coder]

@rektalizerofficial

Of course. Here is params:

Did you forget this?

@ruby_coder

In addition please … a few example lines from your JSONL training file

Thanks

[rektalizerofficial]

Forget what exactly, sorry? Here are few examples:

{"prompt":"How do you respond to criticism? ->","completion":" It really depends on intentions of why I was critisized in the first place. If it's just a troll trying to touch me I either troll back or ignore depending on my mood. If it's a professional criticism that comes from a man that have his brains set right, I will take it proudly and improve on it. That's how you grow, not putting your head in the sand. If it's a questionable piece of criticism, yet still real and have it's points, I will argue to find the truth, which probably lies in between. END"},
{"prompt":"What personality traits does a leader need? ->","completion":" True leader is more or less combination of different valuable personality traits. If we talk about pure high quality top notch leader, it will require so much that people of this caliber being born once in a century. If you talk about leader in general, I think this depends more on what he is about to lead. END"}

[ruby_coder]

Thanks.

With the default 4 n_epochs (underfitted model) and a relatively high temperature of 0.8 it is not surprising you will send a lot of completion variations and randomness.

Hope this helps.

See Also:

Fine-Tuning In a Nutshell with a Single Line JSONL File and n_epochs

Those who frequent this community easily notice that developers struggle to get good fine-tunings results. In this multi-part tutorial, I’m going to use the davinci base model and fine-tune it multiple times with the same single-line prompt using different values for n_epochs. I will take a completion with each of these models, and demonstrate that if you set your n_epochs value high enough, you can get excellent results fine-tuning with a single-line prompt! Before we get started, I would …

[sps]

Thanks for providing the code @rektalizerofficial
Looking at this LoC

@rektalizerofficial

json={'model': MODEL, 'prompt': prompt, 'stop': STOP_SYMBOLS, 'temperature': 0.8, 'max_tokens': 500, 'frequency_penalty': 0.5, "presence_penalty": 0.5}```

Are the variables MODEL, STOP_SYMBOLS, etc global variables? Or have you removed them when sharing the code?

I also cannot find where you’re appending the -> separator to the prompt when calling the API.

[rektalizerofficial]

They are global variables that I define earlier, model is of course my fine-tuned model, and stop sequence is a word END. I did try to both append and not append separator to the prompt, just sent you the latter version. It does help with model to not continue your message, however it still was answering short and sometimes trying to go away from the topic.

[rektalizerofficial]

So if I increase n_epochs value whenever I train it again it will improve?

The main issue for me is that playground version of the same model varies drastically from what I get through API completion. While playground one give a full and rich answer, the API one seems “lazy”.

[sps]

Can you also share the playground version that works for you.

Use the share button in the top-right.