why do you think I want RAG? Is it because once it’s fine-tuned it will lose the chat capabilities and has to be re-fined tuned? This is my assumption given that OpenAI forces you to give the fine-tuned data in chat format/instruction following format.
If I had access to the model I could do DPO to it myself.
Perhaps the best solution would be RAG + fine-tuning on the base model. With OpenAI being closed about what they actually do perhaps it’s not as good as an idea I naively think.