Blog post of April2024: Mid Training

This is part of a recent openai blog:

After testing out prompt engineering, RAG, and fine-tuning, Harvey worked with our team to add the depth of context needed to the model—the equivalent of 10 billion tokens worth of data. Our team modified every step of the model training process, from domain-specific mid-training to customizing post-training processes and incorporating expert attorney feedback.
What is mid-training?
Can we do this? Is it like a first fine tune and then a second based on it? (Other users posts have mentioned that this method is doable but complained that GPT was forgetting…)

I don’t think they’re talking about fine tuning that’s available to you or me. OpenAI seems to be building out its capacity to create custom models, but that’s currently not something that’s available to us mortals :frowning:

I’d love to get an explanation from the OpenAI team though