I am looking to take a complex user input and create a JSON representing that input. This JSON is then used in a separate, independent model which takes a subset (given the JSON data) and intelligently chooses an object from that subset. So, the JSON in the first step of extracting, parameterizing, and normalizing the user input must be quite exact in order to correctly identify the subset.
Currently, I have been using an assistant where, in the instructions, I give a specification of the JSON like so:
"(Some other instructions relevant to my use case blah blah blah). Adhere and conform to the following JSON structure:
{
“stock”: string, (the stock the user specifies, for example $AAPL or $NVDA)
“risk”: int (give a score to the risk of the user input on a 10-point scale)
}
I would give the assistant a concrete input/output example, but I fear I would run up my tokens, for the example would have to be quite large to be meaningful. So: Am I good candidate for fine tuning? If not, why, and what would you suggest? Instinct says yes but it definitely feels like diving into the deep end. I’ve been trying best prompt engineering practices per the docs, but since my user input can be so varied, I’m not sure it’s ever going to work without proper training.
I think fine-tuning may be a feasible option here - subject to testing.
As a base case for the fine-tuning, I would compile a training data set that includes a system message that articulates in more general terms the desired JSON structure as well as other instructions such in relation to how to determine the risk score and then include pairs of user input and assistant responses (i.e. the resulting JSON). With that set-up you would also need to include the system message later on when you consume your fine-tuned model.
This assumes that all information required for the generation of the JSON are provided by the user. If the values in your JSON rely on any other inputs or needs to “translate” user inputs into pre-defined values, then you may still have to combine it with RAG as during the fine-tuning process the model does not actually learn facts. So for example, if the user were to provide the regular company name for the stock and you need to map that name to a specific, pre-defined acronym, then you would need an additional step to achieve that.
Finally, if you do end up fine-tuning, make sure that you create a sufficiently diverse training data set. It sounds like your user inputs can vary quite a bit. Therefore, make sure to also include a sufficient number of edge cases in your data set.
Hey thanks for the well thought-out response! You’re a legend!
Some clarification: I have not yet played with fine tuning. Along with the standard training and validation data, you can also provide a system message? And you are recommending I put more instructions in this system message on how to construct the JSON? And then later on, this system message must be present when hitting the model in production? I do wonder if that somewhat defeats the point of the fine tuning given my now humongous instructions for my assistant (from a $/token perspective). That being said, is there a way to quickly iterate on the system message when it comes to fine tuning? The training data I know exactly what and how to do it, so I’m not too worried there.
Next, RAG: wow! I’m new to this! Reading up on it now, wicked cool. Does OpenAI’s fine tuning have some direct plug-in for this? Or is this just the function calling feature (the “retrieval”)?
I am traveling at the moment so don’t have all the links handy but in short yes you can include a system message in your fine-tuning data. In some cases it is possible to leave the system message out but I think in your case you have some added complexity including due to the risk score. I do believe this requi You can technica instructiostructure itself you can possible condense it further to take advantage of token savings - the model should pick up the logic during the training process.
If you have not already done so, then I would take a look at the fine-tuning guidance to familiarize yourself with the process and requirements for the training data: https://platform.openai.com/docs/guides/fine-tuning
As for RAG: exact set-up depends. Function calling is one option for sure in your case, especially if the user queries are more complex.
You can technically include function calling as part of your fine-tuning. Here’s the link to an OpenAI cookbook illustrating the approach:
Thanks again for your detailed response. I’m looking in to fine-tuning with more granularity now. Are there other options for RAG other than function calling?