Error File is still being processed and is not ready to be used for fine-tuning

API
1

I’m trying to fine-tune a gpt-3.5-turbo model. I’ve used the code github. com/openai/openai-cookbook/blob/main/examples/Chat_finetuning_data_prep.ipynb to validate my dataset (which is basically the toy dataset in the same repository). The result is no error found.

I then uploaded the file via openai.File.create(file=open(file, "rb"), purpose="fine-tune") which returns status uploaded but when I run openai.FineTuningJob.create(training_file=training_file["id"], model="gpt-3.5-turbo") I get

File {} is still being processed and is not ready to be used for fine-tuning. Please try again later.

What should I do to debug this situation? How can I make it working with the toy dataset?

2
  1. Go to “API reference” on the left side bar of this forum.
  2. Read about the fine tuning job object, and its data such as “status”.
  3. Read about “list fine tuning jobs”
  4. Read about “retrieve fine tuning jobs”
  5. Apply this knowledge to check up on when the file becomes ready, which can take many hours.
3
4

The api reference does not say how to move from status uploaded to processed in order to start a job.
I cannot even read the job status when it is created because it throws an exception.

5

There is a queue system. Some have had files pending for a day, or fine-tune jobs also, given that it is again a new buzzword with gpt-3.5 announcement and everyone asking how to do it for the wrong reasons, even though fine tuning has been available for a long time.

You will find the action you need to take in your own first post. Please try again later.

You are right that there is no job status before you’ve even had a tune start, and would get an error. It is the files object you need to list and poll at this point - and you’d get similar status from that as when attempting to start training.

status
The current status of the file, which can be either uploaded, processed, pending, error, deleting or deleted.