Should we finetune only in English?

Hello. Today I successfully tested my first chatGPT3.5 finetuning based on the new documentation. The conversation data I used was not in english, but czech. Since that’s the language we use to speak with the model in our final usecase. But the result seems to just break the model’s responses completely, making it unusable rather than any better. So my question is, since I saw no mention of it anywhere, should we translate our data into english for good results first?

I don’t see any definitive reason why any language can’t be used as training. The same language as all input I would expect to strengthen the reinforcement of responses.

Consider if other recommendations are being used:

  • The system prompt should be included in fine-tuning, should be the same as you are using in your application, and should give the AI a clear new identity to ensure it already recognizes the fine-tuned purpose even before you talk to it.
    Example: “Jste AnswerBot, užitečný AI asistent vyškolený pro Českou republiku Turistickou agenturu.”
    (translation: You are AnswerBot, a helpful AI assistant trained for the Czech Republic Tourism Agency.)

  • An example fine tune should have a sequence [system/user/assistant]. Others are supported, but how they may behave is not documented.

The system prompt being in Czech may have one difference, given that we know that “system” lets us decide on a new fine-tune identity: existing chat tuning will be dominated by “You are a helpful assistant” or “You are ChatGPT” (when trained including a system prompt). Your new system message might de-emphasize prior training even more than if still in English, even when it is “how to chat”.

Ok after a while I managed to improve the results drastically. The key is the Temperature value. While the non-fine-tuned models are doing very well even at default 1, my fine-tuned model has to have this value waay lower to be usable and to actually feel the effect. Maybe obvious for more experienced users, but seems like I am not :grinning:. So if anyone having weird results from fine-tunes, try playing with that before doing anything else.


Yes, that is good information I shared before, but didn’t think of here because of your specific use.

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.