Fine-tuning examples still limited to 4k tokens even on gpt-3.5-turbo-1106

OpenAI Platform the docs mention that the examples for fine-tuning are limited to 4096 tokens is this still true for 16k-context tokens?

File validation would fail on larger example conversation lines.

Pack one with an extra “assistant” role with 4100 tokens of augmentation, and then just nine more placeholders.

I sanitized to 4101 tokens of the AI models list page. Added it as that RAG assistant role before more tokens of system, user, assistant. Whereas a bad json failed, this is in the queue now.

And the fine-tune GUI has no place to chose epochs, so I pay probably 8x more for the little experiment.

1 Like

So lesson: don’t use the GUI to shoot off fine-tune jobs. No suffix. No control over epochs. They could run the thing at your tokens x50 on a whim.


1 Like

Good to know, btw do you know if continuing to finetune a model after the end of a finetuning job will delete the intermediate model? Meaning that if you start a second finetuning job after the end of the second job the intermediate model gets deleted

The first model stays with its original name until you delete it. Just specify that ft:model instead of the original API name model.

1 Like

Nice! But then why am I getting the same model names at least during the second finetuning job?

If you look at the report above, it says base model: gpt-3.5-turbo. I expect that you’d see base model instead as your previous fine tune once underway. Then you receive a new model name when done.

1 Like

Great! Thank you very much for the info :+1:

So anything more than 4k in samples for finetuning doesn’t work?
A bit confused.