You aren’t the only person to have “500” errors trying to tune recently, so it might be something that would have worked before but indeed is acting broken now.
I would try this against the standard gpt-3.5-turbo model (gpt-3.5-turbo-0613).
The preview AI model 1106 will likely have a new version coming out some time soon due to errors reported with its function-call abilities in different languages, giving tunes on the model an uncertain fate. “Recommended” should be changed to “not recommended”.
Also, there have been errors with particular characters, such as accented “e” in training files, but those seemed to be initially caught by a file validation error.
Have you tuned before on similar?
yes, and it worked fine
Are you using functions within the fine-tune? Do they independently validate?
yes, a lot of them. Previously I had trained on the functions
Are you using Unicode or characters outside of ASCII 128?
yes, a few examples had different languages. But now that you had mentioned that Open AI was facing some issues with function calling in different languages - 5 minutes back I just removed all those examples and again started a new job with only ASCII 128 characters.
Are you using all three roles in sequence system, user, assistant? Are you extending that conversation example?
Yes, function is also a role.
Attempting to train on more than 4k tokens per example in total?
yes - a lot of them are above 4k – max it goes to 12k
tried all permutations and combinations after you asked the question – @jr.2509
then divided the main data into small batches and saw which were failing - and then analyzed the data of the failed ones - and then, at last, luckily - got the issue –
it was happening because of some structure error in the arguments of a function call
Open AI is not putting any validation check for these as of now and I think suddenly fine-tuning started expecting only perfect function structures - and that’s what was causing the issues.