During the last months we have trained dozens of fine-tunes on babbage-002 for simple binary classification using training data like:
{"completion":" pos","prompt":"Some text. =>"}
{"completion":" neg","prompt":"Some other text. =>"}
This worked well with high precision values and good recall until now.
Now we trained new fine-tunes in the exact same way like the others before, and these suddenly have zero accuracy, producing always the same token “neg”
The data in these models is exactly the same as in previously trained models (that worked well), and the training process was the same as before (exact same hyperparameters etc.). We also inspected the uploaded training data and it looks exactly the same.
Still, the models now produce only garbage predictions, i.e. always producing “neg” no matter the input.
We tried using davinci, but it led to the exact same result.
Are there any known issues with the fine-tuning process that could cause this? Were there any changes we should be aware of?