Error 'invalid training_file' Fine-Tuning gpt-3.5-turbo-0613

API
1

Hi ! I have prepared my dataset (jsonl) to fine tune gpt-3.5-turbo-0613. My dataset satisfies all criteria posted on openAI website and is marked Ok when validating steps here (Github Chat_finetuning_data_prep.ipynb).
However, I get the error :
error: {
message: ‘invalid training_file: FineTuneSet’,
type: ‘invalid_request_error’,
param: ‘training_file’,
code: null
},

I don’t understand what could be wrong with my jsonl dataset. Here is for example the first line of it :
{“messages”:[{“role”:“system”,“content”:“TEXT.\n TEXT.\n TEXT.”},{“role”:“user”,“content”:“"""TEXT🚀.\r\nTEXT.\r\nTEXT""" \n\n###\n\n”},{“role”:“assistant”,“content”:“TEXT”}]}

Does anyone knows what is going on ? and what does a working dataset must look like ?

2

It looks like you’ve got a mish-mash of old instructions and new instructions there. The stop sequences are not required for chat models, and would actually cause big problems if you trained on that, because the user would have to type ### after every prompt to match up with training.

  1. I would only send line feeds, no \r. That’s not keyboard input.
  2. encode emoji escaped. :rocket: → \u1f680
  3. Is the user really going to type emoji anyway?
  4. Your triple quotes have broken the input. Use single quotes and escape your quotes.
  5. Is the user really going to type triple-quotes anyway?
  6. system prompt should be what your bot will actually use
  7. plain text without extra weird training symbols, unless that’s the actual input and output you want to use.
  8. Every full conversation example: proper JSON on one line.