This is part of a recent openai blog:
After testing out prompt engineering, RAG, and fine-tuning, Harvey worked with our team to add the depth of context needed to the model—the equivalent of 10 billion tokens worth of data. Our team modified every step of the model training process, from domain-specific mid-training to customizing post-training processes and incorporating expert attorney feedback.
What is mid-training?
Can we do this? Is it like a first fine tune and then a second based on it? (Other users posts have mentioned that this method is doable but complained that GPT was forgetting…)
