Unable to upload big JSONL for Vision Model Fine Tuning

API
, ,
1

I am trying to fine tune gpt-4o-2024-08-06 following the guidelines at:

https://platform.openai.com/docs/guides/vision-fine-tuning

I have created JSONL file for training and validation with embedded base64 encoded images.
train.jsonl is around 610 MB.

When fine tuning program starts, it tried to upload the train.jsonl file. But it exits with following error:
————————-
[INFO] OpenAI client initialized with extended timeout.
[INFO] Uploading Training JSONL: data/train_base64.jsonl (610.9 MB)
Starting upload of 610.9MB…
Uploading: 610.9MB / 610.9MB (100.0%)[ERROR] Failed to upload Training after 1 attempts: Error code: 400 - {‘error’: {‘message’: ‘File is too large.’, ‘type’: ‘invalid_request_error’, ‘param’: None, ‘code’: None}}
[ERROR] Aborting due to upload failure.

Is there a limit on the size of JSNOL file? If yes, then what is the limit?

How can we train a model if our dataset has more than 1000 images?

2

I would see if one of the documeted “file size” issues exists first:

  • Image file maximum of 10MB (and then resize - there should be no need to send larger than images reduced so the shorter side is maximum 768 pixels).

  • total ‘example’ part is under 64k tokens.

  • total image count of an entire example is 10 or fewer.

  • max 50k examples with images.

A limit of 512MB rings a bell for certain file inputs, but nothing in documentation currnetly cites that if using purpose:fine-tuning with your upload.