I am building a context-based translator assistant using gpt-4o-mini. I am fine-tuning the gpt-4o-mini, with a custom dataset. In my first attempt i have omitted the system prompt (instructions) from the dataset, and I would send the instructions through the Assistants API. This means that for each request I would need to send the instructions every time (that amounts to token consumption, I understand that recently prompt caching has been introduced).
The instructions include the context (tone of voice, style, guidelines on how to respond, etc.)
Now, I am considering including the system prompt (instructions) in the fine-tuning dataset, so that the instructions will be “baked in” the fine-tuned model, and I wouldn’t need to pass the instructions each time.
Here is what i would like to understand:
Will it actually work? Can the “baked in” system prompt consistently “guide” the model vs sending them on each request?
Should i even consider doing this, considering that the system prompt (instructions) are 350 tokens. (I have slimmed down the instructions from 1,300 tokens to 350 tokens)
Let’s consider a dataset with 10,000 - 20,000 examples, would it be sufficient to include the system prompt for only a smaller batch of examples rather then including it in every example?
You are not “baking in” the system prompt you are essentially training a subset of the model for when your system prompt appears. So, if you then leave your system prompt out you will less likely be navigating to the fine tuned parts of your updated model.
For this to work you might want to include your system prompt in both the training set and the calls.
If on the other hand you create a training set with the system prompt omitted but include the results that were produced with the system prompt in place, that should work.
I thought that maybe i could include the instruction as system prompt in the dataset for 100-500 examples, and that the model would “learn” these instructions, so I wouldn’t need to send them on each request.
Right now i am using the fine-tuned model with the system prompt omitted (the dataset is user: “Translate X from {Source Language} to {Target Language}”, assistant: “Translation”), and I am sending the instructions on each request. It is working fine. I was thinking of shifting the system prompt in the dataset to reduce the token consumption.
So try creating an artificial training set with input omitting the system prompt (whilst including the rest of the original prompt) but output being the results that were achieved when it was included.
I apologise, I wasn’t clear about the dataset. The dataset is generated through scraping a multilingual website for an automotive company. The company has model pages (multilingual so US, UK, France, Germany, Italy, Spain, ect. in total 27 Languages).
The dataset is like this:
user: Translate X (the content of model page US) from English to Italian
assistant: “Translation” (the content of model page Italy).
(The dataset doesn’t include translations generated from the gpt-4o-mini model)
You get the behavior by showing the input and output examples.
Fine-tuning doesn’t work by giving some training examples that have instructions and then the AI magically follows those instructions all the time.
It is by rewarding the model on sequences of input, like if the system prompt says “you have banana brains”, the output is always a number and a stop sequence, that is based on something unpromptable such as the grade level of the writer, thus producing an inference of other such inputs and a mapping based on intelligence.
System message works because of AI pretraining that makes the AI skillful at completing on sentences and ideas, and OpenAI’s post-training on instruction following that still remains after a fine-tune, but you should be considering it as a Manchurian Candidate password that puts your sleeper agent into action - more so than the words themselves having meaning when they are not replicated in usage.
I think we both mean to countermand this kind of thinking:
maybe i could include the instruction as system prompt in the dataset for 100-500 examples, and that the model would “learn” these instructions, so I wouldn’t need to send them on each request.
It is too easy to still think of fine-tuning as teaching the AI model with some kind of dialogue between you and it, just as a forum thread yesterday where it was difficult to communicate that you do not show the AI producing wrong outputs and then the AI writing a correction about what it wrote.
Yes, that is one of the benefits of fine tuning, minimization of system prompt.
However, you have to balance this against a chat model coming with a high level of training already, and how much you want to solicit different results than you’d normally get, or still employ the quality of instruction-following that exists.
If I understand correctly, including the system prompt only occasionally in the dataset won’t help the model “learn”. I’d need to either include it in every example or omit it entirely and send the instructions with each request via the Assistants API.
Since the instructions are just 350 tokens, I’m leaning toward sending them with each request. The current fine-tuned model (trained without the system prompt), works well with a low temperature.
Side question: If I have a dataset of 10,000 translation examples where the output that i am putting, are directly aligned with the brand’s tone and style (taken from the company’s website), does it make sense to completely omit the instructions? Would the model be able to infer the tone and style solely from the dataset?
On my opinion the model would learn how to use the instructions to build the response you provided as the assistant reply. In your calls when the instructions will change it will try to follow the same pattern it was taught in your dataset… I bet the results quality would be lower than expected.
For these type of application I would start by having several sets of instructions with at least 70 or 100 examples per set, then combine everything together and fine-tune on that. So that the goal of fine tuning would be to teach the model how to translate inputs following the provided instructions.
Then I would run this for a while gathering the samples and analyzing the samples to detect the edge cases where your trained model fails, correct the answers manually or using higher level model, and finally, inject them back into the same training set. Then train another model from scratch using the new training set. Then repeat.
