I am trying to fine-tune a model to check code submissions (and grade them). The output must be in JSON format, containing only two keys: “status” and “status_justification.” Even without any fine-tuning the 4o model does really well given the relatively long instructions. However:
When I fine-tune gpt-4o-2024-08-06 with a learning rate of 1e-4, it starts generating lots of extra nonsense that the original 4o model didn’t generate. It even violates the rule of including only the two specified keys in the JSON output, start outputting lots of other ones that were not mentioned anywhere in the instructions.
However, when I fine-tune gpt-4o-mini-2024-07-18 using the same data and hyperparameters, it doesn’t do that. I.e. it doesn’t add extra keys or totally nonsense content to the JSON-output, it behaves pretty much like the unfinetuned model.
Has anyone experienced something similar? Do larger LLMs require significantly smaller learning rates? I couldn’t find an answer to this online.
I have not specified temperature, I guess I am using the default… So I only specified
hyperparameters={
‘n_epochs’: 2,
‘batch_size’: 1,
‘learning_rate_multiplier’: 0.0001}
Note: I only have ~50-100 data points to train on (wanna see if there is any changes at all before increasing the amount of them), that is why the batch_size is just 1.
Thanks for the additional information. I was referring to the temperature setting that you use when consuming the fine-tuned model. One common issue that results in unwanted or even garbled output is to use a too high temperature when consuming the model. Just trying to rule that out as a possible cause.
Also, are you using the same system/user messages as in the training data?
Where I am coming from is that I am trying to rule common issues that can normally impede the functioning of the fine-tuned model. While we should consider the impact oft the learning rate multiplier, I am trying to first eliminate other potential issues.
If you have not set a temperature when using the fine-tuned model and the default value is used, then likely this is not a factor that contributes to the problem. However, to be on the safe side, you might want to consider explicitly setting the temperature with a low value (zero or close to zero).
The question regarding the user / system message was to ensure that you are including the same instructions (whether placed in a system or user message) that you used in your training data when using the fine-tuned model. Naturally, the code would be different but the instructions should be consistent. Stated differently, is there anything you do differently when you use the model compared to what was included in the training data?
What surprises me to hear is that the output format varies when you use your model. That should normally not be the case and I am trying to understand what could be causing this.
@arsenii Hello. It’s important to note that learning_rate_multiplier is a multiplier not a learning rate. Common values for learning_rate_multiplier are e.g. 0.5, 1.0, 2.0, not 1e-4. We set a reasonable learning rate internally (like 1e-4) and then you get to multiply that by some constant to make the model train more or less quickly. Can you try training with learning_rate_multiplier=1.0 and let us know if the model behaves better in that case?
Thank you for replying back and explaining that part! Unfortunately, this is not a solution. I would actually be surprised if that worked since when I was setting a higher learning_multiplier_rate things were getting only worse…
Anyway, I did try setting learning_rate_multiplier=1, but it only made it worse: again the output has literally 50 times more keys that necessary. Remark: for some reason now it is prompt to repeating the same “key”: “value” many times.
@arsenii that is super surprising. Definitely setting a lr multiplier of anything under 0.1 isn’t a realistic solution since that would make the actual learning rate an order of magnitude (or more) below what could actually be useful during training.
Are you using e.g. json_object response format when sampling from the fine-tuned model? Are you using structured outputs? Any other info you can provide?
Would you potentially be open to sharing a (PII redacted / anonymized) version of your dataset with me via email? I’d love to try to debug this and figure out why gpt-4o-2024-08-06 isn’t working well. firstname.lastname@openai.com
Sure, let me try to provide as much info as possible without revealing everything As for “do I use structured outputs”: not the latest addition, only the json one, nothing else.
Overall Goal
Given a solution and some other parameters (e.g. problem statement), output the feedback about this solution given some instructions. The focus is solely on checking if the input solution has any potential to be at least partially valid.
Data For Fine-Tuning
I only have 100 data points (this is preliminary testing before feeding much more data)
The code I use for preparing the file for fine-tuning is
with open('training_data.jsonl', 'w') as f:
for index, row in df.iterrows():
data = {
"messages": [
{"role": "system", "content": instructions},
{"role": "user", "content": row[prompt_col]},
{"role": "assistant", "content": row[expected_response_col]}
]
}
json_string = json.dumps(data)
f.write(json_string + '\n')
training_data_file = self.oai_client.files.create(
file=open('training_data.jsonl', "rb"),
purpose="fine-tune"
)
It is a part of the class I have, but I think it should be self-explanatory.
Fine-Tuning itself
Besides the training file I only give these parameters to the model:
Well, first of all it is surprising that the fine-tuned model outputs something with like 100 keys, when the instructions as well as the training data all have just 2 keys. In particular, for some reason it repeats the same “key”: “value” many many times. Plus it hallucinates a bunch of other stuff.
However, it could be because I made an error somewhere else in the code (some other silly mistake or something). However, when I try all the same (literally the same), but for the gpt-4o-mini-2024-07-18 it works reasonable. By that I mean that at least it does not generate ton of extra nonsense… It might not still be perfect (maybe it is still overfitting or something), but at least it does not do the above.
I have been periodically checking this forum, but every time I missed that message about emailing the dataset etc… I am sorry!
Sadly, the issue is still very much there… and I honestly don’t know what is it related to. Thus I would be happy to share the dataset and/or have a chat elsewhere where I can share more info about what am I doing and what is going wrong. Should I email firstname.lastname@openai.com?
@arsenii please do email firstname.lastname@openai.com with the dataset, I will only use it temporarily to attempt to reproduce the issue and will see if I can help fix this for you.
@john.allard Hi again! I sent the dataset over the weekends, looking forward to hearing back from you! (I think I did not miss any response, but sorry if I did)
Hi, I’ve been having the same issues since release of 4o-fine-tune.
4o-mini fine-tune works like a charm.
When I try to fine-tune 4o, in the dashboard perf/val all look good and better than 4o-mini fine-tune.
But when I try calling the 4o fine-tuned model it just takes forever and produces absolutely nonsenical JSON, similar to what OP has suggested above of random/repeated keys, pure nonsense.
E.g training dataset expects 14 k:v pairs in json. When calling it, it produces 100+.
For calling the model I use temperature 0, json_object, seed=1.
For fine-tuning seed 0, batch 1, lmr 1, epochs 3.
I encounter this always. Once I tried with a very large dataset where output is 40k tokens on avg. And once with a tiny dataset where it probably is 500. All same problem.
I’ve seen some of that behaviour, too, for a recent experimental gpt-4o fine-tune. The issue was not persistent, i.e. in some cases the fine-tuned model worked as intended but 1-2 times it also produced repetitive key-value pairs with hallucinated data.
I also used json_object as output type when this issue occurred.
@john.allard I have not heard from you and the issue remains to be an issue. I think this discussion should NOT be marked as “resolved” In fact, I recently tried fine tuning 4o-mini, and if I put json_object = True, it also produces nonsensical additional keys in the output… I.e. it seems like “json_object” parameter is the real issue-causer here.
As for “do I use structured outputs”: not the latest addition, only the json one, nothing else.
In fact, I recently tried fine tuning 4o-mini, and if I put json_object = True, it also produces nonsensical additional keys in the output… I.e. it seems like “json_object” parameter is the real issue-causer here.