Hi there.
I have gathered a training set for a model, I wish to fine-tune. I have around 80 prompt/completions. Now, I will be gathering training data every week (from my organization), and was wondering if I can “simply” keep fine-tuning my model?
Practically, is it good practice to fine-tune it multiple times? And do I need to keep anything else in mind? (E.g., do I need to include the old/existing training data when re-fine-tuning?)
As far as the docs go:
If you have already fine-tuned a model for your task and now have additional training data that you would like to incorporate, you can continue fine-tuning from the model. This creates a model that has learned from all of the training data without having to re-train from scratch.
To do this, pass in the fine-tuned model name when creating a new fine-tuning job (e.g. -m curie:ft-<org>-<date>). Other training parameters do not have to be changed, however if your new training data is much smaller than your previous training data, you may find it useful to reduce learning_rate_multiplier by a factor of 2 to 4.
When you fine-tune an existing model, do we send a new JSONL with just the new prompt-completions or we add the new prompt-completions to the same previous file (which has all the previous + the new ones) ? I tried with just the new ones, and it seems it completely forgot about the old ones. I also tried adding the old ones to the new one, and the cost it showed me was for the whole file (old + new). What am I missing ?
The quality of continuing on an existing fine tune model as the model name input is all about the learning rate that the passes use.
OpenAI left to its own devices uses a learning multiplier and an epochs number of passes that is high enough to turn gpt-4o into a babbler that can only write like your new small JSONL.
Overtraining to overcome existing overtraining? Yes, but also it is because people using fine-tuning don’t develop ten thousand examples, as you would use for the now non-existent base models. The learning has to be cranked up immediately.
So you will have to use a gentle “touch” if you want the preservation of general chat skills, or for the next round of training to deepen and not dominate. Using hyperparameters, such as epochs:1 to only pay for the training file once, would be a good start.
There are really no “best practices” except anecdote. The only rule is to don’t bother trying to teach it to “talk to your PDFs” or other such knowledge answering.
If you have already fine-tuned a model for your task and now have additional training data that you would like to incorporate, you can continue fine-tuning from the model. This creates a model that has learned from all of the training data without having to re-train from scratch.
I have tried and realized that this is useless! I thought it would give the ability to extand the existing data of the previous fine-tuned model but it re-starts a new other model from scratch. Means the previous dataset of the fine-tuned model (gpt-4o mini) is not included.
That is not how it is supposed to operate.
You have to understand that the AI model doesn’t strictly “contain your data”.
Training increases the weighting favoring predicting sequences similar to your fine-tuning input, affecting behavior, powered by the inference ability of massive pretraining.
This is fine-tuning on 40 examples at OpenAI’s defaults for that training file size, and the AI has lost all prior want to fulfill tasks or even to write longer than examples.
Pass another 40 examples of a pirate through at the same defaults, I’ve got a good feeling that Raven with the goth responses will be gone. Pass the SAME JSON and the weights will be such that only my examples are parroted back.
Thus the need for tuning hyperparameters, because there is nothing in your fine-tuning API request that says “these 10 examples are to be trained at the same learning rate as my 4000 from before, not cranked up like they would need to be to affect gpt-4o trained on 50TB”. You have to take control.
