We have been doing our text completions like this:
Text_from_Whisper=“Good afternoon this is Mary with CVS Pharmacy in Prospect. I’m just calling because I need dr. Scott Van Der his DEA number or NPI number that order to fill a prescription for Michelle Paige Smith date of birth 8 9 74. We would appreciate it. Thank you. By phone number is 203-758-3316. Thank you. Bye.”
Prompt_for_GPT_J=“Extract and display only the first date of birth found as two digits with leading zeros for the month, two digits with leading zeros for the day, and four digits with leading zeros for the year from the following sentence: Text_from_Whisper|DOB:”
Then we get the result.
However we have been doing our fine tunings like this:
Fine_tuning_rule=“Good afternoon this is Mary with CVS Pharmacy in Prospect. I’m just calling because I need dr. Scott Van Der his DEA number or NPI number that order to fill a prescription for Michelle Paige Smith date of birth 8 9 74. We would appreciate it. Thank you. By phone number is 203-758-3316. Thank you. Bye.|DOB: 08/09/1974”
So the question is should our fine tunings actually be like this instead?
Fine_tuning_rule=“Extract and display only the first date of birth found as two digits with leading zeros for the month, two digits with leading zeros for the day, and four digits with leading zeros for the year from the following sentence: Good afternoon this is Mary with CVS Pharmacy in Prospect. I’m just calling because I need dr. Scott Van Der his DEA number or NPI number that order to fill a prescription for Michelle Paige Smith date of birth 8 9 74. We would appreciate it. Thank you. By phone number is 203-758-3316. Thank you. Bye.|DOB: 08/09/1974”
Or is it ok to just do them like this?
Fine_tuning_rule=“Good afternoon this is Mary with CVS Pharmacy in Prospect. I’m just calling because I need dr. Scott Van Der his DEA number or NPI number that order to fill a prescription for Michelle Paige Smith date of birth 8 9 74. We would appreciate it. Thank you. By phone number is 203-758-3316. Thank you. Bye.|DOB: 08/09/1974”
According to my understanding, fine-tuning, among other purposes, is done to reduce the prompt tokens to reduce cost for a high volume of calls.
In my opinion, it should be just the text from Whisper as prompt and the desired result in completion. Even remove the term “DOB:” from the completion.
Well, if you think about it… the model would suddenly have the same exact text 100 to 1000 times depending on your dataset. So, it would be “baked in” that those sequences of words are important (because they showed up so often in the training data…) and that could affect your output when querying the fine-tuned model.
It is generally recommended to include the prompt in the fine-tuning rule to ensure that the model learns to generate the desired output based on the input. Therefore, it would be better to include the prompt in the fine-tuning rule, as in the following example:
Fine_tuning_rule=“Extract and display only the first date of birth found as two digits with leading zeros for the month, two digits with leading zeros for the day, and four digits with leading zeros for the year from the following sentence: Good afternoon this is Mary with CVS Pharmacy in Prospect. I’m just calling because I need dr. Scott Van Der his DEA number or NPI number that order to fill a prescription for Michelle Paige Smith date of birth 8 9 74. We would appreciate it. Thank you. By phone number is 203-758-3316. Thank you. Bye.|DOB: 08/09/1974”
I mean, ChatGPT is known to give false information, but sure.
This is from the docs…
In particular, while prompts for base models often consist of multiple examples (“few-shot learning”), for fine-tuning, each training example generally consists of a single input example and its associated output, without the need to give detailed instructions or include multiple examples in the same prompt.
Might be best to try it both ways and see which resulting model gives best results for your use-case?
I think it’s a good idea when we use a common vocabulary to talk about technology, especially when coding solutions @silvacarl; and to be honest, your use of the term “Fine_tuning_rule” makes it more time consuming to process your questions and provide guidance.
Are you talking about GPT-J or OpenAI fine-tunings?
If you are talking about OpenAI fine-tunings, how does your use of the term “Fine_tuning_rule” map to a JSONL line, exactly? Can you post an example of your JSONL file?
we are testing both. we need to do 10K calls a day, and the issue is that openai runs out of tokens per day and we have not been able to get them to increase it enough for our use case.
so we are working on fine tuning of both and comparing results.
So yes, we are doing testing on both openAI and GPT-J-6B side by side fine tuned to beanchmark each one.
What did you find out? I have a large dataset of regulatory information. I’ve broken down into sections, and would like to fine-tune with a multi-question prompt for each completion section. Did you try this method and did it work?
