Is it possible to Fine-Tuning OpenAI model with Reinforcement Learning from Human Feedback ? If yes, how I can do that ?
Definitely something I would like to know about as well. From what I can see currently, weāre obliged to re-train our custom model with the added prompt-completions (the full JSONL prompt-completions).
Yes, ultimately thatās all you have available to you with current fine-tune models, the inputs/outputs you train it on, and the evaluation of the final product. And start again if you donāt like it.
Held-out validation datasets, weights and biases, training continuation, basically anything to see how well the model performs donāt seem to be available. OpenAI: āthe best solution to poor results is to turn off the results.ā
What can you do with an OpenAI model? Gather high temperature varied results of your tuning for user inputs (where 1.0 is now high), send them off to be human-evaluated, pick the best (and send āall badā off for human answering), retrain on more of the best.
Hereās an OpenAI article on RLHF LLM model alignment, just the basic overview and a touch of what the training inputs can do. And you can immediately see what you canāt do. Aligning language models to follow instructions
I had considered doing something akin to what you described: using 100 sets of data to fine tune for instance, then getting it to generate a 101st, then fixing the output of that, then reusing that as the 101st training sample, however I considered that 101st training data set will be 90% ai generated. Will that result in data contamination and thus performance loss?
Thatās my only worry, I could understand how it wouldnāt because youāre just improving the aiās answer, not giving it new pre training data, but Iām wondering where the line is there?