I am trying to fine tuning a model to exhibit certain type of behave. I want to use reinforcement fine tuning method to complete the fine tuning.
As of now Open AI provides only o4-mini for reinforcement fine tuning.
Is there any alternative to complete the fine tuning job ?
No more model fine-tunes: Note the deprecations page in the developer documentation site: fine-tuning itself will be over and dead by the end of the year, as well as the o4-mini model being shut off. Fine tuning is completely shut off on your org if you haven’t used fine tuning before.
Reinforcement fine-tuning is needed where OpenAI doesn’t let you directly train the reasoning AI models on reasoning portions of context you constructed yourself, and where you aren’t given the internal proprietary form of the language to even make reasoning. You merely get that the “answer” that results from actually running the model in tests is promoted or discouraged as the way the learning takes place, specific to reasoning models.
Your title suggests gpt-4.1 series models, which are non-reasoning. You can directly use supervised on those models, and receive results that are more controllable. DPO is kind a similar concept of learning based on positive/negative reinforcement pairs.
So I would complete the “job” by writing a compatible supervised training set, for gpt-4.1. That will give an inference model with the longest lifespan.
Thank you so much for the detailed explanation, _j! This really cleared things up.
Just to add a bit of context, this is part of a research project Im working on, and I was exploring RFT as an alternative approach to see if it could yield different results for my use case. Your note about RFT being specifically designed around the inaccessibility of reasoning traces is really insightful and helps me understand why it may not be the right fit here.
I’ll continue exploring the supervised fine-tuning path for gpt-4.1. Really appreciate you taking the time.