Does GPT-4o lose ability to accept images once it has been fine-tuned

API
, ,
1

I’ve recently been experimenting with the fine-tuning capabilities of the GPT-4o model. While I understand that it’s currently not possible to include images in the fine-tuning data for GPT-4o, I was surprised to find that the fine-tuned model lost its ability to accept images altogether.

When including base64 image in the prompt the fine-tuned model responds with Invalid content type. image_url is only supported by certain models.

Did anyone experience similar issues, or did anyone manage to send image to the fine-tuned GPT-4o model?

2 Likes
2

The fine-tuned model is known to lose its vision capabilities.

Personally, I think this should be clearly stated in the documentation…

3 Likes
3

Yes, currently fine-tuning is only supported for text modality.

Thus when you fine-tune, vision modality is disabled for the model.

2 Likes
4

Yeah, I’ve seen this happen before. Fine-tuning GPT-4 seems to disable its ability to handle images since that part isn’t included in the process. Sticking to the base model is probably your best bet if you’re working with images. You’re not the only one running into this issue!